Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

A quick first pass at a new iteration of the realtime diffusion pipeline: physical interface → camera → realtime img2img → upscale → 3DGS through the new TripoSplat. Rough and held together with tape for now, but the whole loop runs end to end. Next: - better input and... show more

Hugues Bruyère

8,074 subscribers

15,526 просмотров • 4 месяцев назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

This is crazy, I can talk to OpenAI's realtime API, and it'll respond back with voice, and then help call Codex to help me code! Got curious about Codex and its realtime integration, and found codex's feature flag `realtime_conversation` integrated for the codex app-server. How it works: - launch codex app-server with realtime - connect a client to realtime through WebRTC voice connection - Codex app-server is also attached on the side - talk to realtime - realtime decides whether coding is needed, and responds back with events - Codex app-server takes these events and runs codex agent turns - codex writes code like your codex cli exp Keep in mind OpenAI hasn't released this officially, they're still actively working on it, I just created a fork. This is a completely new UX than the usual chat interface for agentic coding. For example, when the agent responds back in voice, it's actually faster for me to read the response, and I don't actually find voice response that useful unless it does so in the background. But image realtime working in the background, you can talk to it, realtime keeps track of the different agents working on your tasks and your status. More to come on demo side, i'll make a more polished version and include voice, but thought this was cool to share. You can see in the responses the distinction between my voice input, realtime response, and realtime calling Codex to do work!

This is crazy, I can talk to OpenAI's realtime API, and it'll respond back with voice, and then help call Codex to help me code! Got curious about Codex and its realtime integration, and found codex's feature flag `realtime_conversation` integrated for the codex app-server. How it works: - launch codex app-server with realtime - connect a client to realtime through WebRTC voice connection - Codex app-server is also attached on the side - talk to realtime - realtime decides whether coding is needed, and responds back with events - Codex app-server takes these events and runs codex agent turns - codex writes code like your codex cli exp Keep in mind OpenAI hasn't released this officially, they're still actively working on it, I just created a fork. This is a completely new UX than the usual chat interface for agentic coding. For example, when the agent responds back in voice, it's actually faster for me to read the response, and I don't actually find voice response that useful unless it does so in the background. But image realtime working in the background, you can talk to it, realtime keeps track of the different agents working on your tasks and your status. More to come on demo side, i'll make a more polished version and include voice, but thought this was cool to share. You can see in the responses the distinction between my voice input, realtime response, and realtime calling Codex to do work!

Matthew Lam

17,674 просмотров • 4 месяцев назад

Introducing GPT-Realtime-2 in the API: our most intelligent voice model yet, bringing GPT-5-class reasoning to voice agents. Voice agents are now real-time collaborators that can listen, reason, and solve complex problems as conversations unfold. Now available in the API alongside streaming models GPT-Realtime-Translate and GPT-Realtime-Whisper — a new set of audio capabilities for the next generation of voice interfaces.

Introducing GPT-Realtime-2 in the API: our most intelligent voice model yet, bringing GPT-5-class reasoning to voice agents. Voice agents are now real-time collaborators that can listen, reason, and solve complex problems as conversations unfold. Now available in the API alongside streaming models GPT-Realtime-Translate and GPT-Realtime-Whisper — a new set of audio capabilities for the next generation of voice interfaces.

OpenAI

3,668,761 просмотров • 5 месяцев назад

Painting with 3D Gaussian Splat Brushes Contributions: • Designing a set of interactive tools and brush parameters for artistic brush creation and control using 3DGS content. • Computing oriented 3D Gaussian splat brushes for stamp-based painting on 3D surfaces represented as meshes or 3DGS scenes. • Deforming the splats in a brush stamp to ensure a smooth appearance of the painted 3D stroke. • Producing seamless brush strokes despite overlapping brush stamps, using diffusion inpainting. • Efficient modeling and rendering of brush strokes to facilitate 3DGS painting in real time.

Painting with 3D Gaussian Splat Brushes Contributions: • Designing a set of interactive tools and brush parameters for artistic brush creation and control using 3DGS content. • Computing oriented 3D Gaussian splat brushes for stamp-based painting on 3D surfaces represented as meshes or 3DGS scenes. • Deforming the splats in a brush stamp to ensure a smooth appearance of the painted 3D stroke. • Producing seamless brush strokes despite overlapping brush stamps, using diffusion inpainting. • Efficient modeling and rendering of brush strokes to facilitate 3DGS painting in real time.

MrNeRF

23,060 просмотров • 1 год назад

3DGS Render Addon for Blender is out! New features include: - Paint and use image textures to edit colour - Mesh to 3DGS conversion - Transformations and edits are exportable - Lot's of other boosts and little features! Link for the addon and a Quick Guide tutorial below #b3d #blender #3dgs #gaussiansplatting

3DGS Render Addon for Blender is out! New features include: - Paint and use image textures to edit colour - Mesh to 3DGS conversion - Transformations and edits are exportable - Lot's of other boosts and little features! Link for the addon and a Quick Guide tutorial below #b3d #blender #3dgs #gaussiansplatting

KIRI Engine - 3D Scanner App

19,019 просмотров • 1 год назад

the input interface has been the same for decades. with ai, software can now reason and act on your behalf but the interface is the bottleneck. why do i have to check sushi on 10 restaurants across 3 apps? why can't i just do it with a flick of a finger?! the world's about to get a new interface @agi_interfaces

the input interface has been the same for decades. with ai, software can now reason and act on your behalf but the interface is the bottleneck. why do i have to check sushi on 10 restaurants across 3 apps? why can't i just do it with a flick of a finger?! the world's about to get a new interface @agi_interfaces

sahil dhull

89,024 просмотров • 5 месяцев назад

Realtime interactive generative models FTW! Announcing a new 🌊 of details and features for Magenta RealTime, the open weights live music AI model from GDM! * Live Jamming with audio input 🎤🎸🎵 * Personalize your own models 🔧 * Tech report 📜 Links below in the 🧵...

Realtime interactive generative models FTW! Announcing a new 🌊 of details and features for Magenta RealTime, the open weights live music AI model from GDM! * Live Jamming with audio input 🎤🎸🎵 * Personalize your own models 🔧 * Tech report 📜 Links below in the 🧵...

Jesse Engel

183,802 просмотров • 1 год назад

In the past few weeks, I deep dived into an exploration revolving around the use of physical interfaces to feed and interact with a real-time img2img diffusion pipeline using Stream Diffusion and SDXL Turbo. What really captivated me is to use my hands, objects, art supplies, tools, and light to create images and scenes. 𝗣𝗵𝘆𝘀𝗶𝗰𝗮𝗹 𝗠𝗲𝗱𝗶𝗮 + 𝗧𝗼𝗼𝗹𝘀 I experimented with clay, manipulating different types and colors along with a selection of prompts. I used a magnifying glass, tracked in real-time, to focus the diffusion process on specific areas. Combining these tools created a dynamic and inspiring experience. Using magic clay to layer shapes and colors as a base for revealing landscapes and hidden worlds, and the magnifying glass to focus and reveal these details, was particularly effective. 𝗣𝗵𝘆𝘀𝗶𝗰𝗮𝗹 𝗟𝗶𝗴𝗵𝘁 I used light as my method of interaction with the img2img diffusion. This approach felt special right away. There was something magical about holding a physical light source and seeing it influence the generated visuals. I iterated on this technique with themes like Rococo architecture, flowers, Brutalist architecture, hidden worlds, and origami landscapes. 𝗜𝗻𝗸 + 𝗛𝘆𝗯𝗿𝗶𝗱 𝗙𝗼𝗿𝗺𝗮𝘁𝘀 I also used ink in milk as a means of physical interaction with the diffusion pipeline. As I drop ink into milk, shapes come alive instantly. By learning to manipulate the combination of physical and digital elements, I steered the generated output toward my areas of interest. These iterations extended beyond ink in milk to include the format in which these elements are contained: a circular plate or a triptych of small stainless steel trays. These formats provide a structured yet flexible framework to explore themes and narratives across multiple visual spaces. It's magical. Some of that last iteration has been captured in this insightful article by Fast Company: #stablediffusion #realtime #ai

In the past few weeks, I deep dived into an exploration revolving around the use of physical interfaces to feed and interact with a real-time img2img diffusion pipeline using Stream Diffusion and SDXL Turbo. What really captivated me is to use my hands, objects, art supplies, tools, and light to create images and scenes. 𝗣𝗵𝘆𝘀𝗶𝗰𝗮𝗹 𝗠𝗲𝗱𝗶𝗮 + 𝗧𝗼𝗼𝗹𝘀 I experimented with clay, manipulating different types and colors along with a selection of prompts. I used a magnifying glass, tracked in real-time, to focus the diffusion process on specific areas. Combining these tools created a dynamic and inspiring experience. Using magic clay to layer shapes and colors as a base for revealing landscapes and hidden worlds, and the magnifying glass to focus and reveal these details, was particularly effective. 𝗣𝗵𝘆𝘀𝗶𝗰𝗮𝗹 𝗟𝗶𝗴𝗵𝘁 I used light as my method of interaction with the img2img diffusion. This approach felt special right away. There was something magical about holding a physical light source and seeing it influence the generated visuals. I iterated on this technique with themes like Rococo architecture, flowers, Brutalist architecture, hidden worlds, and origami landscapes. 𝗜𝗻𝗸 + 𝗛𝘆𝗯𝗿𝗶𝗱 𝗙𝗼𝗿𝗺𝗮𝘁𝘀 I also used ink in milk as a means of physical interaction with the diffusion pipeline. As I drop ink into milk, shapes come alive instantly. By learning to manipulate the combination of physical and digital elements, I steered the generated output toward my areas of interest. These iterations extended beyond ink in milk to include the format in which these elements are contained: a circular plate or a triptych of small stainless steel trays. These formats provide a structured yet flexible framework to explore themes and narratives across multiple visual spaces. It's magical. Some of that last iteration has been captured in this insightful article by Fast Company: #stablediffusion #realtime #ai

Hugues Bruyère

86,381 просмотров • 2 лет назад

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior paper page: present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation.

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior paper page: present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation.

AK

161,530 просмотров • 3 лет назад

Voice-only programming with the new OpenAI Realtime API ... I spend a lot of time these days pair programming with LLMs. Often I'm talking rather than typing. This "voice dictation" use case has become an important vibe benchmark for me. Being able to create text input just by talking, flexibly, in a context dependent way, with tool calling, is a *hard* problem for today's models. Natural language dictation requires a very high degree of contextual intelligence, instruction following accuracy, and tool calling reliability. Today's new gpt-realtime model is quite good at this hard problem. The original realtime model release last year was impressive. Seeing what a speech-to-speech model could do got a lot of people excited about the possibilities of voice AI. The improvements since that first release are equally impressive. I can use this new model, now, for real world tasks that were past the edge of the "jagged frontier" before. Here's a video showing a couple of fun (and tricky) modes of voice input.

Voice-only programming with the new OpenAI Realtime API ... I spend a lot of time these days pair programming with LLMs. Often I'm talking rather than typing. This "voice dictation" use case has become an important vibe benchmark for me. Being able to create text input just by talking, flexibly, in a context dependent way, with tool calling, is a hard problem for today's models. Natural language dictation requires a very high degree of contextual intelligence, instruction following accuracy, and tool calling reliability. Today's new gpt-realtime model is quite good at this hard problem. The original realtime model release last year was impressive. Seeing what a speech-to-speech model could do got a lot of people excited about the possibilities of voice AI. The improvements since that first release are equally impressive. I can use this new model, now, for real world tasks that were past the edge of the "jagged frontier" before. Here's a video showing a couple of fun (and tricky) modes of voice input.

kwindla

50,874 просмотров • 1 год назад

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution paper page: Text-based diffusion models have exhibited remarkable success in generation and editing, showing great promise for enhancing visual content with their generative prior. However, applying these models to video super-resolution remains challenging due to the high demands for output fidelity and temporal consistency, which is complicated by the inherent randomness in diffusion models. Our study introduces Upscale-A-Video, a text-guided latent diffusion framework for video upscaling. This framework ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences; globally, without training, a flow-guided recurrent latent propagation module is introduced to enhance overall video stability by propagating and fusing latent across the entire sequences. Thanks to the diffusion paradigm, our model also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation, enabling a trade-off between fidelity and quality. Extensive experiments show that Upscale-A-Video surpasses existing methods in both synthetic and real-world benchmarks, as well as in AI-generated videos, showcasing impressive visual realism and temporal consistency.

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution paper page: Text-based diffusion models have exhibited remarkable success in generation and editing, showing great promise for enhancing visual content with their generative prior. However, applying these models to video super-resolution remains challenging due to the high demands for output fidelity and temporal consistency, which is complicated by the inherent randomness in diffusion models. Our study introduces Upscale-A-Video, a text-guided latent diffusion framework for video upscaling. This framework ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences; globally, without training, a flow-guided recurrent latent propagation module is introduced to enhance overall video stability by propagating and fusing latent across the entire sequences. Thanks to the diffusion paradigm, our model also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation, enabling a trade-off between fidelity and quality. Extensive experiments show that Upscale-A-Video surpasses existing methods in both synthetic and real-world benchmarks, as well as in AI-generated videos, showcasing impressive visual realism and temporal consistency.

AK

32,849 просмотров • 2 лет назад

The Oil Market Is Disconnected From The Physical 🔊 On Friday's edition of The Macro Show, Hedgeye Asset Management's R. Patrick Kent R. Patrick Kent (RPK) and Ryan Ricci Ryan Ricci explained why a down day in oil is not the all-clear the tape treats it as. As Ricci noted, the number one reason something falls is that it just rose 20% to 30% in a straight line, not a change in the geopolitics. Crude is still closer to the top end of its risk range than the low end, and prediction markets put the odds of Strait of Hormuz shipping traffic returning to normal by year-end at roughly 18%. RPK added two points: 1. The futures market is disconnected from the physical, where Brent priced across Dubai and Shanghai sits around $120 to $125 a barrel, a gap that closes when the contracts roll at month-end. 2. The disruption has spread well beyond the Strait. The East-West pipeline is running at half capacity at best for months, Red Sea traffic runs through Houthi-controlled waters, and moving barrels on small boats shepherded by US destroyers carries at least $20 a barrel in incremental cost. "I'll take the under on the 20% chance of it being normalized by the end of the year." That leaves the question of what normal even means now, and how far off it still is. Subscribe to The Macro Show for daily market analysis from Hedgeye:

The Oil Market Is Disconnected From The Physical 🔊 On Friday's edition of The Macro Show, Hedgeye Asset Management's R. Patrick Kent R. Patrick Kent (RPK) and Ryan Ricci Ryan Ricci explained why a down day in oil is not the all-clear the tape treats it as. As Ricci noted, the number one reason something falls is that it just rose 20% to 30% in a straight line, not a change in the geopolitics. Crude is still closer to the top end of its risk range than the low end, and prediction markets put the odds of Strait of Hormuz shipping traffic returning to normal by year-end at roughly 18%. RPK added two points: 1. The futures market is disconnected from the physical, where Brent priced across Dubai and Shanghai sits around $120 to $125 a barrel, a gap that closes when the contracts roll at month-end. 2. The disruption has spread well beyond the Strait. The East-West pipeline is running at half capacity at best for months, Red Sea traffic runs through Houthi-controlled waters, and moving barrels on small boats shepherded by US destroyers carries at least $20 a barrel in incremental cost. "I'll take the under on the 20% chance of it being normalized by the end of the year." That leaves the question of what normal even means now, and how far off it still is. Subscribe to The Macro Show for daily market analysis from Hedgeye:

Hedgeye

15,051 просмотров • 26 дней назад

I've been talking about these real-time AI render layers for a few years now as an obvious next step for 3d, but seeing it unfolding (albeit still in the early stages), is pretty amazing. This is Krea's new model that transforms your input in near-realtime (I adjusted the slight offset a bit to make it more obvious here). I'll post an un-edited video in the replies.

I've been talking about these real-time AI render layers for a few years now as an obvious next step for 3d, but seeing it unfolding (albeit still in the early stages), is pretty amazing. This is Krea's new model that transforms your input in near-realtime (I adjusted the slight offset a bit to make it more obvious here). I'll post an un-edited video in the replies.

Martin Nebelong

66,625 просмотров • 1 год назад

📣 New research from GenAI at Meta, introducing Meta 3D Gen: A new system for end-to-end generation of 3D assets from text in <1min. Meta 3D Gen is a new combined AI system that can generate high-quality 3D assets, with both high-resolution textures and material maps end-to-end, producing results that are superior to existing solutions — at 3-10x the speed of existing work in this space. Details in the technical report ➡️

📣 New research from GenAI at Meta, introducing Meta 3D Gen: A new system for end-to-end generation of 3D assets from text in <1min. Meta 3D Gen is a new combined AI system that can generate high-quality 3D assets, with both high-resolution textures and material maps end-to-end, producing results that are superior to existing solutions — at 3-10x the speed of existing work in this space. Details in the technical report ➡️

AI at Meta

409,091 просмотров • 2 лет назад

Americans overwhelmingly want the Iran war to end, and many see hope for lower gas prices now. But a new CBS News poll finds relatively few Americans think the U.S. is getting the better of the new agreement with Iran.

Americans overwhelmingly want the Iran war to end, and many see hope for lower gas prices now. But a new CBS News poll finds relatively few Americans think the U.S. is getting the better of the new agreement with Iran.

Face The Nation

12,365 просмотров • 3 месяцев назад

Not the flashiest demos, but what’s under the hood represents a foundational shift for general-purpose robotics. World models are the next-gen foundation of Physical AI, not the VLM backbones found in typical VLAs. DreamZero is a 14B-parameter World Action Model (WAM) by NVIDIA that treats robotics as a joint video-and-action prediction task. Unlike traditional Vision-Language-Action (VLA) models that map images directly to motor commands, DreamZero leverages a pretrained video diffusion backbone to predict future world states and actions simultaneously. - achieves 2× better zero-shot generalization to unseen tasks and environments compared to state-of-the-art VLAs. - learns effectively from heterogeneous, non-repetitive data (500 hours), breaking the need for thousands of repeated demonstrations. - adapts to new robot embodiments with just 30 minutes of play data. - enables 7Hz closed-loop control via system optimizations and "DreamZero-Flash," making high-capacity diffusion models viable for real-time use.

Not the flashiest demos, but what’s under the hood represents a foundational shift for general-purpose robotics. World models are the next-gen foundation of Physical AI, not the VLM backbones found in typical VLAs. DreamZero is a 14B-parameter World Action Model (WAM) by NVIDIA that treats robotics as a joint video-and-action prediction task. Unlike traditional Vision-Language-Action (VLA) models that map images directly to motor commands, DreamZero leverages a pretrained video diffusion backbone to predict future world states and actions simultaneously. - achieves 2× better zero-shot generalization to unseen tasks and environments compared to state-of-the-art VLAs. - learns effectively from heterogeneous, non-repetitive data (500 hours), breaking the need for thousands of repeated demonstrations. - adapts to new robot embodiments with just 30 minutes of play data. - enables 7Hz closed-loop control via system optimizations and "DreamZero-Flash," making high-capacity diffusion models viable for real-time use.

The Humanoid Hub

35,364 просмотров • 8 месяцев назад

MVDream: Multi-view Diffusion for 3D Generation paper page: propose MVDream, a multi-view diffusion model that is able to generate geometrically consistent multi-view images from a given text prompt. By leveraging image diffusion models pre-trained on large-scale web datasets and a multi-view dataset rendered from 3D assets, the resulting multi-view diffusion model can achieve both the generalizability of 2D diffusion and the consistency of 3D data. Such a model can thus be applied as a multi-view prior for 3D generation via Score Distillation Sampling, where it greatly improves the stability of existing 2D-lifting methods by solving the 3D consistency problem. Finally, we show that the multi-view diffusion model can also be fine-tuned under a few shot setting for personalized 3D generation, i.e. DreamBooth3D application, where the consistency can be maintained after learning the subject identity.

MVDream: Multi-view Diffusion for 3D Generation paper page: propose MVDream, a multi-view diffusion model that is able to generate geometrically consistent multi-view images from a given text prompt. By leveraging image diffusion models pre-trained on large-scale web datasets and a multi-view dataset rendered from 3D assets, the resulting multi-view diffusion model can achieve both the generalizability of 2D diffusion and the consistency of 3D data. Such a model can thus be applied as a multi-view prior for 3D generation via Score Distillation Sampling, where it greatly improves the stability of existing 2D-lifting methods by solving the 3D consistency problem. Finally, we show that the multi-view diffusion model can also be fine-tuned under a few shot setting for personalized 3D generation, i.e. DreamBooth3D application, where the consistency can be maintained after learning the subject identity.

AK

294,519 просмотров • 3 лет назад

Just a little taste of the Robohacks hackathon happening now at Y Combinator in San Francisco. This is only two of the teams. The first is a team building a robot for an AirBnb on Mars. The second built a system that 3D prints new tools for the robot to do new work on Mars. Both demonstrate why I love builders so much. Dreaming of a new world and learning together how to build for a new robot none of them have used before.

Just a little taste of the Robohacks hackathon happening now at Y Combinator in San Francisco. This is only two of the teams. The first is a team building a robot for an AirBnb on Mars. The second built a system that 3D prints new tools for the robot to do new work on Mars. Both demonstrate why I love builders so much. Dreaming of a new world and learning together how to build for a new robot none of them have used before.

Robert Scoble

14,399 просмотров • 6 месяцев назад