Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

🚀🎬We introduce TMD (Transition Matching Distillation): 480p videos generated from text prompts in < 3 NFEs! 1️⃣Main backbone for feature extraction and lightweight head for iterative refinement 2️⃣Distilled from Wan2.1 14B T2V combining MeanFlow & DMD2 🔗

julberner's profile picture

Julius Berner

1,320 subscribers

16,985 Aufrufe • vor 7 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

TurboEdit Instant text-based image editing discuss: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.
0:22

TurboEdit Instant text-based image editing discuss: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.

AK

16,062 Aufrufe • vor 2 Jahren

A 19-euro tool just replaced 8,000 Meta employees. Here’s the exact workflow a TikTok creator used to build AI influencer videos from scratch — no camera, no team, no budget. Step 1. Open Pinterest. Search “Girl Self.” Save the image you like. Open Nanobanana Pro. Step 2. Swap the face using a single prompt. (Link at the bottom.) Step 3. Take your generated photo into Crust. Currently the easiest video-from-image tool on the market. Step 4. Create 3 nodes: reference image, text, video generator. Step 5. Drop your photo into the image node. Write a video prompt into the text field. This is what makes the photo move. Step 6. Connect the nodes. Text to text-in. Image to image-in. Select Kling 2.6. Set duration. Toggle sound. Step 7. Hit generate. 2 prompts. 1 AI girl. 0 employees. Meta spent $14B on Reality Labs last year to build digital humans. This creator built one in 7 minutes for less than a dinner. The prompts are free in the reply.
0:45

A 19-euro tool just replaced 8,000 Meta employees. Here’s the exact workflow a TikTok creator used to build AI influencer videos from scratch — no camera, no team, no budget. Step 1. Open Pinterest. Search “Girl Self.” Save the image you like. Open Nanobanana Pro. Step 2. Swap the face using a single prompt. (Link at the bottom.) Step 3. Take your generated photo into Crust. Currently the easiest video-from-image tool on the market. Step 4. Create 3 nodes: reference image, text, video generator. Step 5. Drop your photo into the image node. Write a video prompt into the text field. This is what makes the photo move. Step 6. Connect the nodes. Text to text-in. Image to image-in. Select Kling 2.6. Set duration. Toggle sound. Step 7. Hit generate. 2 prompts. 1 AI girl. 0 employees. Meta spent $14B on Reality Labs last year to build digital humans. This creator built one in 7 minutes for less than a dinner. The prompts are free in the reply.

RGK

187,070 Aufrufe • vor 2 Monaten

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.
1:43

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.

AK

40,485 Aufrufe • vor 2 Jahren

As detailed in the Meta Movie Gen technical report, today we’re open sourcing Movie Gen Bench: two new media generation benchmarks that we hope will help to enable the AI research community to progress work on more capable audio and video generation models. Movie Gen Video Bench is the largest and most comprehensive benchmark ever released for evaluating text-to-video generation. It includes a collection of 1,000+ prompts that cover concepts ranging from detailed human activity to animals, physics, unusual subjects and more — with broad coverage across different motion levels. Movie Gen Audio Bench is a first-of-its-kind benchmark aimed at evaluating video-to-audio and (text+video)-to-audio generation. It includes 527 generated videos and associated sound effects and music prompts covering a diverse set of ambient environments and sound effects. To enable fair and easy comparison to our models for future works, these new benchmarks include non cherry-picked generated videos and audio from Movie Gen. In releasing these new benchmarks we hope to promote fair & extensive evaluations in media generation research to enable greater progress in this field.
0:23

As detailed in the Meta Movie Gen technical report, today we’re open sourcing Movie Gen Bench: two new media generation benchmarks that we hope will help to enable the AI research community to progress work on more capable audio and video generation models. Movie Gen Video Bench is the largest and most comprehensive benchmark ever released for evaluating text-to-video generation. It includes a collection of 1,000+ prompts that cover concepts ranging from detailed human activity to animals, physics, unusual subjects and more — with broad coverage across different motion levels. Movie Gen Audio Bench is a first-of-its-kind benchmark aimed at evaluating video-to-audio and (text+video)-to-audio generation. It includes 527 generated videos and associated sound effects and music prompts covering a diverse set of ambient environments and sound effects. To enable fair and easy comparison to our models for future works, these new benchmarks include non cherry-picked generated videos and audio from Movie Gen. In releasing these new benchmarks we hope to promote fair & extensive evaluations in media generation research to enable greater progress in this field.

AI at Meta

156,273 Aufrufe • vor 1 Jahr

Is VideoGen starting to become good enough for robotic manipulation? 🤖 Check out our recent work, RIGVid — Robots Imitating Generated Videos — where we use AI-generated videos as intermediate representations and 6-DoF motion retargeting to guide robots in diverse manipulation tasks: pouring, wiping, mixing, and more. 🔗 Key takeaways: - VideoGen starts to become good enough for robotics - As the field progresses, we are expecting much better results in the coming years - Depending on whether video prediction models take actions or not (VideoGen vs Action-Conditioned Video Prediction), there are different ways to use them. - Controllability & steerability are still issues In the paper, we explore: – How do different video generation models compare for robotic imitation? – Can generated videos replace real videos for imitation? – What causes failure of imitation given high-quality videos? – How does imitating from video compare with other representations (e.g., keypoint constraints like ReKep)? 🎥 Watch the video for (1) AI-generated inputs, (2) robot executions, and (3) the 3D intermediate representation bridging the embodiment gap.
0:39

Is VideoGen starting to become good enough for robotic manipulation? 🤖 Check out our recent work, RIGVid — Robots Imitating Generated Videos — where we use AI-generated videos as intermediate representations and 6-DoF motion retargeting to guide robots in diverse manipulation tasks: pouring, wiping, mixing, and more. 🔗 Key takeaways: - VideoGen starts to become good enough for robotics - As the field progresses, we are expecting much better results in the coming years - Depending on whether video prediction models take actions or not (VideoGen vs Action-Conditioned Video Prediction), there are different ways to use them. - Controllability & steerability are still issues In the paper, we explore: – How do different video generation models compare for robotic imitation? – Can generated videos replace real videos for imitation? – What causes failure of imitation given high-quality videos? – How does imitating from video compare with other representations (e.g., keypoint constraints like ReKep)? 🎥 Watch the video for (1) AI-generated inputs, (2) robot executions, and (3) the 3D intermediate representation bridging the embodiment gap.

Yunzhu Li

16,540 Aufrufe • vor 1 Jahr

The future is here. 🚀 Introducing Nwiro | Your Unreal Engine AI Assistant, a new AI powered PCG environment creation tool by Leartes. nwiro.ai Turn a simple text prompt into fully generated Unreal Engine environments in minutes. Intelligent asset placement. Automated PCG workflows. Faster, smarter world building. From idea to immersive world, instantly. This is an Alpha release, and we are just getting started. Download Nwiro and get 3 free credits to start creating right away. Create beyond limits. Create with Nwiro. 🌍✨ Get Nwiro on Fab for free and give feedback to unlock more features! 🔗 🔗 Check out the Nwiro website #Nwiro #Leartes #UnrealEngine #PCG #GameDev #AI #WorldBuilding #Tool
1:36

The future is here. 🚀 Introducing Nwiro | Your Unreal Engine AI Assistant, a new AI powered PCG environment creation tool by Leartes. nwiro.ai Turn a simple text prompt into fully generated Unreal Engine environments in minutes. Intelligent asset placement. Automated PCG workflows. Faster, smarter world building. From idea to immersive world, instantly. This is an Alpha release, and we are just getting started. Download Nwiro and get 3 free credits to start creating right away. Create beyond limits. Create with Nwiro. 🌍✨ Get Nwiro on Fab for free and give feedback to unlock more features! 🔗 🔗 Check out the Nwiro website #Nwiro #Leartes #UnrealEngine #PCG #GameDev #AI #WorldBuilding #Tool

Leartes

69,194 Aufrufe • vor 6 Monaten

The “I Believe” music video has officially landed on Joe’s YouTube channel for the very first time — now upscaled in HD! 🎬🎸 A standout track from Flying in a Blue Dream, “I Believe” is also one of the rare songs to feature Joe on vocals 🎤 Directed by David Hogan, who got his start designing legendary album covers for Stax Records and went on to direct Barb Wire (1996) and also work on Batman Forever and Alien 3. Watch now via the link in bio or at and stand by as we have news of more classic videos getting a good upscale treatment! 🎞️ #JoeSatriani #FlyingInABlueDream
0:53

The “I Believe” music video has officially landed on Joe’s YouTube channel for the very first time — now upscaled in HD! 🎬🎸 A standout track from Flying in a Blue Dream, “I Believe” is also one of the rare songs to feature Joe on vocals 🎤 Directed by David Hogan, who got his start designing legendary album covers for Stax Records and went on to direct Barb Wire (1996) and also work on Batman Forever and Alien 3. Watch now via the link in bio or at and stand by as we have news of more classic videos getting a good upscale treatment! 🎞️ #JoeSatriani #FlyingInABlueDream

Joe Satriani

17,007 Aufrufe • vor 2 Monaten

XAI EXTENDS GROK IMAGINE VIDEO GENERATION TO 10 SECONDS WITH QUALITY ENHANCEMENTS xAI has updated its Grok Imagine tool to produce videos lasting 10 seconds, doubling the prior limit. This change, along with refinements in visual and audio elements, expands the tool's utility for short-form content creation. xAI released the upgrade to Grok Imagine in early 2026. The company, founded by Elon Musk, announced the feature through posts on the X platform. This follows previous iterations where videos were capped at shorter durations, typically 5 seconds. Grok Imagine allows users to generate videos from text prompts, building on its image creation capabilities. The update addresses constraints in video length that limited expressive potential. Users can now input descriptions to create clips, such as animations or scenes, without needing initial images. This positions the tool within the broader landscape of AI-driven multimodal generation, where text-to-video systems are increasingly common. The core adjustment doubles the maximum video duration from 5 seconds to 10 seconds. Accompanying this are upgrades to video quality, including more stable visuals, richer details, and improved clarity. Audio has also been enhanced for better output, making the generated content more immersive. These changes were described as "big improvements across the board" in the announcement. No specific benchmarks or quantitative metrics for the quality improvements were detailed in the release statements. The feature rollout appears gradual, with some users accessing it via the Grok app or web interface. xAI has not introduced user controls for exact timing, though such options are mentioned as future possibilities. This development highlights xAI's emphasis on iterative enhancements in generative AI tools. By extending duration while refining output fidelity, it reflects engineering priorities aimed at balancing computational efficiency with user needs. The focus on audio and visual stability suggests attention to common pitfalls in early text-to-video models, such as inconsistencies or artifacts. The sources do not specify the underlying model architecture changes or training data adjustments enabling this upgrade. Performance in real-world scenarios, like handling complex prompts or maintaining consistency across clips, remains unquantified in the announcements. Interpretations of broader implications for AI video generation would require additional evidence beyond what's provided.
0:29

XAI EXTENDS GROK IMAGINE VIDEO GENERATION TO 10 SECONDS WITH QUALITY ENHANCEMENTS xAI has updated its Grok Imagine tool to produce videos lasting 10 seconds, doubling the prior limit. This change, along with refinements in visual and audio elements, expands the tool's utility for short-form content creation. xAI released the upgrade to Grok Imagine in early 2026. The company, founded by Elon Musk, announced the feature through posts on the X platform. This follows previous iterations where videos were capped at shorter durations, typically 5 seconds. Grok Imagine allows users to generate videos from text prompts, building on its image creation capabilities. The update addresses constraints in video length that limited expressive potential. Users can now input descriptions to create clips, such as animations or scenes, without needing initial images. This positions the tool within the broader landscape of AI-driven multimodal generation, where text-to-video systems are increasingly common. The core adjustment doubles the maximum video duration from 5 seconds to 10 seconds. Accompanying this are upgrades to video quality, including more stable visuals, richer details, and improved clarity. Audio has also been enhanced for better output, making the generated content more immersive. These changes were described as "big improvements across the board" in the announcement. No specific benchmarks or quantitative metrics for the quality improvements were detailed in the release statements. The feature rollout appears gradual, with some users accessing it via the Grok app or web interface. xAI has not introduced user controls for exact timing, though such options are mentioned as future possibilities. This development highlights xAI's emphasis on iterative enhancements in generative AI tools. By extending duration while refining output fidelity, it reflects engineering priorities aimed at balancing computational efficiency with user needs. The focus on audio and visual stability suggests attention to common pitfalls in early text-to-video models, such as inconsistencies or artifacts. The sources do not specify the underlying model architecture changes or training data adjustments enabling this upgrade. Performance in real-world scenarios, like handling complex prompts or maintaining consistency across clips, remains unquantified in the announcements. Interpretations of broader implications for AI video generation would require additional evidence beyond what's provided.

Lacey

28,881 Aufrufe • vor 7 Monaten

Guys, I need your help! How is X protecting original content creators? They aren't. In fact, the Head of Product blocks them! Here's my original video about the Tower of Babel: my idea for the comparison, my prompts, my credits, and my original text accompanying the video. It received 1.5 million impressions on X. Three days later, people stole my video, added their own watermarks, and posted it on TikTok, making money from my work. They even translated my entire text into their native language and reposted it. The result? I received a 90% reduction in revenue share, while this guy is probably making money from the 200,000 views generated by my work. Meanwhile, the same Head of Product is protecting content thieves on X who steal videos and simply add their own watermarks. How is this fair? Will someone from X answer me? If anyone has connections to X, Elon, or anyone else at the company, please ask them. I'm not crying about the low revenue share. I'm upset because there is no justice. Original creators are punished, while content thieves are rewarded.
0:37

Guys, I need your help! How is X protecting original content creators? They aren't. In fact, the Head of Product blocks them! Here's my original video about the Tower of Babel: my idea for the comparison, my prompts, my credits, and my original text accompanying the video. It received 1.5 million impressions on X. Three days later, people stole my video, added their own watermarks, and posted it on TikTok, making money from my work. They even translated my entire text into their native language and reposted it. The result? I received a 90% reduction in revenue share, while this guy is probably making money from the 200,000 views generated by my work. Meanwhile, the same Head of Product is protecting content thieves on X who steal videos and simply add their own watermarks. How is this fair? Will someone from X answer me? If anyone has connections to X, Elon, or anyone else at the company, please ask them. I'm not crying about the low revenue share. I'm upset because there is no justice. Original creators are punished, while content thieves are rewarded.

Open Minded Approach

64,558 Aufrufe • vor 3 Monaten

🎥 Today we’re premiering Meta Movie Gen: the most advanced media foundation models to-date. Developed by AI research teams at Meta, Movie Gen delivers state-of-the-art results across a range of capabilities. We’re excited for the potential of this line of research to usher in entirely new possibilities for casual creators and creative professionals alike. More details and examples of what Movie Gen can do ➡️ 🛠️ Movie Gen models and capabilities Movie Gen Video: 30B parameter transformer model that can generate high-quality and high-definition images and videos from a single text prompt. Movie Gen Audio: A 13B parameter transformer model that can take a video input along with optional text prompts for controllability to generate high-fidelity audio synced to the video. It can generate ambient sound, instrumental background music and foley sound — delivering state-of-the-art results in audio quality, video-to-audio alignment and text-to-audio alignment. Precise video editing: Using a generated or existing video and accompanying text instructions as an input it can perform localized edits such as adding, removing or replacing elements — or global changes like background or style changes. Personalized videos: Using an image of a person and a text prompt, the model can generate a video with state-of-the-art results on character preservation and natural movement in video. We’re continuing to work closely with creative professionals from across the field to integrate their feedback as we work towards a potential release. We look forward to sharing more on this work and the creative possibilities it will enable in the future.
0:30

🎥 Today we’re premiering Meta Movie Gen: the most advanced media foundation models to-date. Developed by AI research teams at Meta, Movie Gen delivers state-of-the-art results across a range of capabilities. We’re excited for the potential of this line of research to usher in entirely new possibilities for casual creators and creative professionals alike. More details and examples of what Movie Gen can do ➡️ 🛠️ Movie Gen models and capabilities Movie Gen Video: 30B parameter transformer model that can generate high-quality and high-definition images and videos from a single text prompt. Movie Gen Audio: A 13B parameter transformer model that can take a video input along with optional text prompts for controllability to generate high-fidelity audio synced to the video. It can generate ambient sound, instrumental background music and foley sound — delivering state-of-the-art results in audio quality, video-to-audio alignment and text-to-audio alignment. Precise video editing: Using a generated or existing video and accompanying text instructions as an input it can perform localized edits such as adding, removing or replacing elements — or global changes like background or style changes. Personalized videos: Using an image of a person and a text prompt, the model can generate a video with state-of-the-art results on character preservation and natural movement in video. We’re continuing to work closely with creative professionals from across the field to integrate their feedback as we work towards a potential release. We look forward to sharing more on this work and the creative possibilities it will enable in the future.

AI at Meta

2,265,850 Aufrufe • vor 1 Jahr

🚀 SANA-Video 2.0 is here! A full-stack optimized video model designed for efficiency — while still delivering high quality. We combine a hybrid architecture closely related to the recent Kimi K3 design, adopt Self-Flow from FLUX 3, and further accelerate it with our own Sol-Engine. Key technical ingredients: 🧠 Hybrid Linear–Softmax Attention 3 gated linear layers + 1 gated-softmax anchor (75% linear / 25% softmax) 🧱 Block Attention Residuals (AttnRes) Boosts deep-layer effective rank by ~12% 🏗️ Unified 5B & 14B models Trained from scratch under very limited resources: • 5B → only 16 nodes of H100 • 14B → 48 nodes of B200 Results 📊 • 84.30 VBench Total • 3.2× faster DiT forward than matched full-softmax at 720p/60s • 720p/5s in 13.06s on a single H100 with Sol-Engine • 120× faster than Wan 2.2-A14B under the same one-H100 setup SANA-Video 2.0 shows that high-quality 720p video generation can be both efficient and practical on a single GPU. 🎬 Project: 📄 Paper: 💻 Code: Proud of the team! 🎉 More details below 🧵
0:24

🚀 SANA-Video 2.0 is here! A full-stack optimized video model designed for efficiency — while still delivering high quality. We combine a hybrid architecture closely related to the recent Kimi K3 design, adopt Self-Flow from FLUX 3, and further accelerate it with our own Sol-Engine. Key technical ingredients: 🧠 Hybrid Linear–Softmax Attention 3 gated linear layers + 1 gated-softmax anchor (75% linear / 25% softmax) 🧱 Block Attention Residuals (AttnRes) Boosts deep-layer effective rank by ~12% 🏗️ Unified 5B & 14B models Trained from scratch under very limited resources: • 5B → only 16 nodes of H100 • 14B → 48 nodes of B200 Results 📊 • 84.30 VBench Total • 3.2× faster DiT forward than matched full-softmax at 720p/60s • 720p/5s in 13.06s on a single H100 with Sol-Engine • 120× faster than Wan 2.2-A14B under the same one-H100 setup SANA-Video 2.0 shows that high-quality 720p video generation can be both efficient and practical on a single GPU. 🎬 Project: 📄 Paper: 💻 Code: Proud of the team! 🎉 More details below 🧵

Enze Xie

25,338 Aufrufe • vor 1 Monat

Our team S.N. Bose National Centre for Basic Sciences introduces PathGennie (just published in JCIM & JCTC Journals), a fast and unbiased computational framework for uncovering rare molecular events such as drug unbinding and protein folding. Instead of using any biased forces or elevated temperatures, PathGennie launches ultrashort unbiased MD trajectories and selectively propagates only those that show real progress toward the transition. This lets us generate complete rare-event pathways in just 10–100 ps, while preserving true molecular dynamics. Full credit goes to my PhD student Dibyendu Maity DIBYENDU MAITY for formulating and implementing the PathGennie code. Thanks to Anusandhan National Research Foundation for funding. ✨ Key Highlights: • Identified 7 distinct unbinding routes for benzene from T4 lysozyme • Recovered all major exit pathways for imatinib from Abl kinase within a handful of attempts • Captured folding/unfolding pathways of Trp-cage and Protein G that align with minimum free-energy paths • Weighted Ensemble (WE) simulations converge 3–4× faster when initialised with PathGennie-generated pathways • Path finding works even with suboptimal CVs 🔗 Try our code: 📄 Read the paper: #CompChem #RareEvent #EnhancedSampling #MolecularDynamics #PathSampling #DrugDiscovery
0:27

Our team S.N. Bose National Centre for Basic Sciences introduces PathGennie (just published in JCIM & JCTC Journals), a fast and unbiased computational framework for uncovering rare molecular events such as drug unbinding and protein folding. Instead of using any biased forces or elevated temperatures, PathGennie launches ultrashort unbiased MD trajectories and selectively propagates only those that show real progress toward the transition. This lets us generate complete rare-event pathways in just 10–100 ps, while preserving true molecular dynamics. Full credit goes to my PhD student Dibyendu Maity DIBYENDU MAITY for formulating and implementing the PathGennie code. Thanks to Anusandhan National Research Foundation for funding. ✨ Key Highlights: • Identified 7 distinct unbinding routes for benzene from T4 lysozyme • Recovered all major exit pathways for imatinib from Abl kinase within a handful of attempts • Captured folding/unfolding pathways of Trp-cage and Protein G that align with minimum free-energy paths • Weighted Ensemble (WE) simulations converge 3–4× faster when initialised with PathGennie-generated pathways • Path finding works even with suboptimal CVs 🔗 Try our code: 📄 Read the paper: #CompChem #RareEvent #EnhancedSampling #MolecularDynamics #PathSampling #DrugDiscovery

Suman Chakrabarty

54,164 Aufrufe • vor 9 Monaten

[SIGGRAPH 2025] Photoreal Scene Reconstruction from an Egocentric Device Contributions: 1. We address the importance of employing visual-inertial bundle adjustment (VIBA) that accounts for the rolling-shutter behavior of the RGB camera. This provides a continuous camera trajectory to model pixel movement in neural reconstruction. Our experiments demonstrate that using VIBA consistently improves the novel view quality in Gaussian Splatting by +1 dB in PSNR. 2. We introduce a rasterization-based image formulation pipeline that addresses common artifacts in physical image formation, including rolling shutter, lens shading, exposure, and gain compensation. Our approach is distinct in that we represent image poses as posed pixel arrays sampled from a continuous trajectory, rather than assigning a single camera pose per image, and preserve the merit of Gaussian rasterization. Unlike existing methods that require ray-tracing Gaussians, e.g., [Moenne-Loccoz et al. 2024], our formulation is applicable to general-purpose rasterization-based Gaussian splatting. When applied to 3D Gaussian Splatting (3DGS) [Kerbl et al. 2023], our approach can further enhance reconstruction quality by +1 dB. We outperform existing baselines and demonstrate a substantial quality improvement in handling complex scenes observed by egocentric devices. 3. To reduce the effect of blur from rapid head motion in darker indoor scenes, we propose a strategy of deliberately underexposing input videos during capture, inspired by HDR+ [Hasinoff et al. 2016]. We demonstrate that we can reconstruct high-quality, noise-free scene radiance from noisy, dim input videos, and further render sharp, blur-free videos at a higher dynamic range.
1:00

[SIGGRAPH 2025] Photoreal Scene Reconstruction from an Egocentric Device Contributions: 1. We address the importance of employing visual-inertial bundle adjustment (VIBA) that accounts for the rolling-shutter behavior of the RGB camera. This provides a continuous camera trajectory to model pixel movement in neural reconstruction. Our experiments demonstrate that using VIBA consistently improves the novel view quality in Gaussian Splatting by +1 dB in PSNR. 2. We introduce a rasterization-based image formulation pipeline that addresses common artifacts in physical image formation, including rolling shutter, lens shading, exposure, and gain compensation. Our approach is distinct in that we represent image poses as posed pixel arrays sampled from a continuous trajectory, rather than assigning a single camera pose per image, and preserve the merit of Gaussian rasterization. Unlike existing methods that require ray-tracing Gaussians, e.g., [Moenne-Loccoz et al. 2024], our formulation is applicable to general-purpose rasterization-based Gaussian splatting. When applied to 3D Gaussian Splatting (3DGS) [Kerbl et al. 2023], our approach can further enhance reconstruction quality by +1 dB. We outperform existing baselines and demonstrate a substantial quality improvement in handling complex scenes observed by egocentric devices. 3. To reduce the effect of blur from rapid head motion in darker indoor scenes, we propose a strategy of deliberately underexposing input videos during capture, inspired by HDR+ [Hasinoff et al. 2016]. We demonstrate that we can reconstruct high-quality, noise-free scene radiance from noisy, dim input videos, and further render sharp, blur-free videos at a higher dynamic range.

MrNeRF

15,244 Aufrufe • vor 1 Jahr

Today we're announcing #GAIA1: a 9B parameter world model, trained on 4,700 hours of driving data, able to simulate complex and diverse driving scenes from video, text and action inputs. This model is 480x larger than the preview we shared earlier this year and the results are incredible. These videos are entirely synthetically generated by Wayve's generative AI, GAIA-1. But there is more here than just generating videos, GAIA is an entire world model. A world model allows us to simulate the future, conditioned on video, text and action inputs, which can be leveraged for making informed decisions when driving. Why is this game-changing for autonomous driving? 1. Safety. One limitation with AI systems like today's Large Language Models is that they are autoregressive, next-word prediction algorithms, but aren't necessarily aware of the implications of their decisions. A world model allows us to give our AI the capability to be aware of its decisions, by simulating the future, which is important for self-driving safety. 2. Synthetic training data. I believe synthetic training data is the future for AI, because it is safer, cheaper, and infinitely scalable. GAIA-1 unlocks unprecedented realism and diversity of synthetic data for self-driving. 3. Long-tail robustness. One of the biggest challenges for self-driving is long-tail robustness: dealing with the enormous magnitude of edge cases we see on the road. An advantage of generative AI is its incredible ability to recombine experiences in new ways. This is exciting for self-driving as it means we can learn from two edge case scenarios, and combine them to become a corner case. For example, we can experience driving in fog, and experience of jay-walking pedestrians, and GAIA can learn from these experiences to understand how to generate a fog+jay walking scenario. Check out many more videos in our blog or further technical details in our paper: Or come chat with our team who are at the International Conference on Computer Vision (#ICCV2023) this week in Paris in Booth 32 Jamie Shotton
0:41

Today we're announcing #GAIA1: a 9B parameter world model, trained on 4,700 hours of driving data, able to simulate complex and diverse driving scenes from video, text and action inputs. This model is 480x larger than the preview we shared earlier this year and the results are incredible. These videos are entirely synthetically generated by Wayve's generative AI, GAIA-1. But there is more here than just generating videos, GAIA is an entire world model. A world model allows us to simulate the future, conditioned on video, text and action inputs, which can be leveraged for making informed decisions when driving. Why is this game-changing for autonomous driving? 1. Safety. One limitation with AI systems like today's Large Language Models is that they are autoregressive, next-word prediction algorithms, but aren't necessarily aware of the implications of their decisions. A world model allows us to give our AI the capability to be aware of its decisions, by simulating the future, which is important for self-driving safety. 2. Synthetic training data. I believe synthetic training data is the future for AI, because it is safer, cheaper, and infinitely scalable. GAIA-1 unlocks unprecedented realism and diversity of synthetic data for self-driving. 3. Long-tail robustness. One of the biggest challenges for self-driving is long-tail robustness: dealing with the enormous magnitude of edge cases we see on the road. An advantage of generative AI is its incredible ability to recombine experiences in new ways. This is exciting for self-driving as it means we can learn from two edge case scenarios, and combine them to become a corner case. For example, we can experience driving in fog, and experience of jay-walking pedestrians, and GAIA can learn from these experiences to understand how to generate a fog+jay walking scenario. Check out many more videos in our blog or further technical details in our paper: Or come chat with our team who are at the International Conference on Computer Vision (#ICCV2023) this week in Paris in Booth 32 Jamie Shotton

Alex Kendall

631,869 Aufrufe • vor 2 Jahren

great to see more people generating 3d avatars with our new text-to-3d feature in forge. this marks a step in the right direction in putting powerful creation tools directly in the hands of everyone. we built forge entirely from the ground up over the past months as one of the key releases on our roadmap. having full proprietary ownership of the technology gives us complete control to shape its direction without depending on external platforms or third-party licenses. building on this foundation, our upcoming studio feature will let users generate high-quality accessories, clothing, and environments simply by typing natural language prompts. a single description can produce fully textured, production-ready 3d assets in seconds, with options to create multiple variations and refine them through follow-up instructions.every asset created in studio integrates seamlessly with the 3d ai agents made in forge on users can instantly apply clothing and accessories with automatic fitting, layer multiple items, and place their agents inside custom-generated environments. all clothing and accessories come pre-rigged and optimized, while environments include proper lighting and geometry for immediate use in games, animation, virtual worlds, and more. we have spent months thoughtfully designing how can deliver real, sustainable value back to the community. we will continue to share more details on token utility use cases and the economic flywheel we have built. the goal is to create a self-reinforcing system where creators earn through royalties, autonomous agents drive on-chain activity, and platform growth directly benefits active community members and token holders. together, these features enable a complete creative flow. from a simple idea, anyone can quickly build fully realized 3d ai agents standing in rich, custom scenes. we are excited to see what the community builds next.
0:24

great to see more people generating 3d avatars with our new text-to-3d feature in forge. this marks a step in the right direction in putting powerful creation tools directly in the hands of everyone. we built forge entirely from the ground up over the past months as one of the key releases on our roadmap. having full proprietary ownership of the technology gives us complete control to shape its direction without depending on external platforms or third-party licenses. building on this foundation, our upcoming studio feature will let users generate high-quality accessories, clothing, and environments simply by typing natural language prompts. a single description can produce fully textured, production-ready 3d assets in seconds, with options to create multiple variations and refine them through follow-up instructions.every asset created in studio integrates seamlessly with the 3d ai agents made in forge on users can instantly apply clothing and accessories with automatic fitting, layer multiple items, and place their agents inside custom-generated environments. all clothing and accessories come pre-rigged and optimized, while environments include proper lighting and geometry for immediate use in games, animation, virtual worlds, and more. we have spent months thoughtfully designing how can deliver real, sustainable value back to the community. we will continue to share more details on token utility use cases and the economic flywheel we have built. the goal is to create a self-reinforcing system where creators earn through royalties, autonomous agents drive on-chain activity, and platform growth directly benefits active community members and token holders. together, these features enable a complete creative flow. from a simple idea, anyone can quickly build fully realized 3d ai agents standing in rich, custom scenes. we are excited to see what the community builds next.

nich

19,611 Aufrufe • vor 2 Monaten

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!
0:30

Want to create an avatar from a single image? FlexAvatar is a transformer model that creates full 360°, high-quality, and expressive 3D head avatar from just a single portrait image in minutes. Real-time Demo: FlexAvatar's lightweight architecture allows both animation and rendering in real-time, enabling interactive user experiences. To create a new 3D head avatar, only one image is required, e.g., from a webcam. The final avatar is ready after 2 minutes. Architecture: Under the hood, FlexAvatar adopts a transformer-based encoder-decoder design. The encoder maps the input image onto a latent avatar space, while the decoder produces 3D Gaussian attribute maps by incorporating the animation signal via cross-attention. The model learns all facial animations directly from the data without relying on pre-built 3D face models. This equips the avatars with realistic facial expressions. The internal avatar latent space can be conveniently used to integrate additional observations of a person via fitting. This enables use-cases where more than one image of a person is available, e.g., from a phone scan of the person. We train jointly on 2D monocular videos and multi-view data. However, in monocular videos, the animation signal leaks the target viewpoint, causing the model to produce incomplete 3D heads. We call this phenomenon entanglement of driving signal and target viewpoint. To prevent entanglement, we introduce bias sinks. These are learnable tokens that indicate whether a training sample stems from a monocular or a multi-view dataset. During training, the model learns to produce incomplete 3D heads only when the monocular token is present. During inference, FlexAvatar then always uses the multi-view token for which the model has learned to produce complete 3D heads. This simple design allows to combine the generalizability from monocular data with the quality of multi-view data. FlexAvatar summary: - Input: Single-image, phone scan, or monocular video - Output: Full 360° head avatar - Expressive animations - Real-time rendering and animation - Generalization to any portrait - Create a new avatar in 2 minutes - Use bias sinks to combine 2D and 3D data 🏠 🌍 🎥 Great work by Tobias Kirschstein and Simon Giebenhain!

Matthias Niessner

96,238 Aufrufe • vor 8 Monaten

New short course: Vibe Coding 101 with Replit! Learn to build and host applications with an AI agent in this course, built in partnership with Replit ⠕ and taught by its President Michele Catasta and Head of Developer Relations . Coding agents are changing how we write code. "Vibe coding" refers to a growing practice where you might barely look at the generated code, and instead focus on the architecture and features of your application. However, contrary to popular belief, effectively coding this way isn't done by just prompting, accepting all recommendations, and hoping for the best. It requires structuring your work, refining your prompts, and having a systematic process that lead to a more efficient and effective workflow. I code frequently using LLMs, and asking an LLM to do everything in one shot usually does not work. I'll typically take a problem, partition it into manageable modules, spend time creating prompts to specify each module, and use the model to produce the code one module at a time, and test/debug each module before moving on. A process like this is making me and many other developers faster and more efficient. In this video-only course, you’ll learn how to use Replit’s cloud environment--with an integrated code editor, package manager, and deployment tools--to build and deploy web applications. Along the way, you’ll learn strategies for working effectively with agents and improve your development skills. In detail, you’ll: - Understand principles of agentic code development such as being precise, giving agents one task at a time, making prompts specific, keeping projects tidy, starting with fresh sessions for each new feature, and how to approach debugging. - Learn how to get started with Replit, and key skills for vibe coding: Thinking, using frameworks, checkpoints, debugging, and providing context. - Create a product requirement document (PRD) and wireframe for your agent to build a prototype of a website performance analyzer. - See how to use an agent to make your prototype more visually appealing, and deploy it application others to access . - Learn to build a head-to-head national park ranking app, from a sample dataset, with voting capabilities and persistent data storage, and refine further ask the assistant to recap and explain what it built to find room for improvement and reinforce your learning. By the end of this course, you’ll have a solid foundation in building with coding agents, and a process you can use to keep vibe coding effectively. Please sign up here:
2:10

New short course: Vibe Coding 101 with Replit! Learn to build and host applications with an AI agent in this course, built in partnership with Replit ⠕ and taught by its President Michele Catasta and Head of Developer Relations . Coding agents are changing how we write code. "Vibe coding" refers to a growing practice where you might barely look at the generated code, and instead focus on the architecture and features of your application. However, contrary to popular belief, effectively coding this way isn't done by just prompting, accepting all recommendations, and hoping for the best. It requires structuring your work, refining your prompts, and having a systematic process that lead to a more efficient and effective workflow. I code frequently using LLMs, and asking an LLM to do everything in one shot usually does not work. I'll typically take a problem, partition it into manageable modules, spend time creating prompts to specify each module, and use the model to produce the code one module at a time, and test/debug each module before moving on. A process like this is making me and many other developers faster and more efficient. In this video-only course, you’ll learn how to use Replit’s cloud environment--with an integrated code editor, package manager, and deployment tools--to build and deploy web applications. Along the way, you’ll learn strategies for working effectively with agents and improve your development skills. In detail, you’ll: - Understand principles of agentic code development such as being precise, giving agents one task at a time, making prompts specific, keeping projects tidy, starting with fresh sessions for each new feature, and how to approach debugging. - Learn how to get started with Replit, and key skills for vibe coding: Thinking, using frameworks, checkpoints, debugging, and providing context. - Create a product requirement document (PRD) and wireframe for your agent to build a prototype of a website performance analyzer. - See how to use an agent to make your prototype more visually appealing, and deploy it application others to access . - Learn to build a head-to-head national park ranking app, from a sample dataset, with voting capabilities and persistent data storage, and refine further ask the assistant to recap and explain what it built to find room for improvement and reinforce your learning. By the end of this course, you’ll have a solid foundation in building with coding agents, and a process you can use to keep vibe coding effectively. Please sign up here:

Andrew Ng

752,900 Aufrufe • vor 1 Jahr

Sora 2 has the capabilities to make an entire anime episode with time and effort. This is a turning point for generative AI models and it is terrifying to think what is next to come. The difference between Sora and Sora 2 is absolutely staggering. Yes, this looks cool. I put the time into it to make it as close to art as possible. This was not simple to make and not just prompting and throwing it in some editor. Every frame is generated of another scene and even the reference image for characters I used to start this project were AI generated. This was all created with 10 second clips pulled apart and chunked out into the final product. Regardless of the imagery you see, I did not directly use any artist's art for this video. This 10-minute video was generated over the course of a week from more than 700 text prompts. It was built on technology trained by scraping the uncredited, uncompensated contributions of countless human artists and animators. The creation of this single video consumed an enormous amount of energy and water, equivalent to powering a home for days and requiring hundreds of liters of fresh water for cooling. But that is nothing. In the 10 minutes you spend watching this, the global network of AI video generators will create over 6,250 more short videos. The combined energy required for that 10 minutes of global creation is enough to power an average household for nearly 2 years. - It's a double-edged sword. While AI uses immense amount of energy has clear immoral issues with scalping the hard work of artists, this also provides those a medium who have potentially spent their life attempting to draw out the ideas in their head and failing to grasp it. My sister is an artist, I grew up always attempting to draw but never could get the image in my head on paper, I've spent the better part of over 20 years to teach myself through watching videos and practice, however I don't have steady hands and frankly have just been unable to make any vision come to life. Which brings me to the conflict here. This would have been a dream of mine to be able to get frames created and flesh out the story in my head. To make this video professionally, it would be an absurd amount of money. A small studio by itself would be over a million dollars just to start up. To hire a studio would be most likely over ~200k. I would love to make this story I have in my head through legit and traditional means, I would love to start a Kickstarter to get funding and hire artists, voice actors, and a production team. However I know that would most likely be an impossible reach that would further fuel the hate. Regardless, I understand the worry this causes. The fear this produces. However, we can't just ignore where AI is currently at and just tell people not to use it or even worse threaten people who use it. People will always use the shiny new toy in front of them, so the real question is how do we either make it work for us and work along side it, or how do we ACTUALLY implement a method to protect art rather than tell people to not use it. Respect your artists. Review their ToS and don't upload their hard work to a model without permission.
10:33

Sora 2 has the capabilities to make an entire anime episode with time and effort. This is a turning point for generative AI models and it is terrifying to think what is next to come. The difference between Sora and Sora 2 is absolutely staggering. Yes, this looks cool. I put the time into it to make it as close to art as possible. This was not simple to make and not just prompting and throwing it in some editor. Every frame is generated of another scene and even the reference image for characters I used to start this project were AI generated. This was all created with 10 second clips pulled apart and chunked out into the final product. Regardless of the imagery you see, I did not directly use any artist's art for this video. This 10-minute video was generated over the course of a week from more than 700 text prompts. It was built on technology trained by scraping the uncredited, uncompensated contributions of countless human artists and animators. The creation of this single video consumed an enormous amount of energy and water, equivalent to powering a home for days and requiring hundreds of liters of fresh water for cooling. But that is nothing. In the 10 minutes you spend watching this, the global network of AI video generators will create over 6,250 more short videos. The combined energy required for that 10 minutes of global creation is enough to power an average household for nearly 2 years. - It's a double-edged sword. While AI uses immense amount of energy has clear immoral issues with scalping the hard work of artists, this also provides those a medium who have potentially spent their life attempting to draw out the ideas in their head and failing to grasp it. My sister is an artist, I grew up always attempting to draw but never could get the image in my head on paper, I've spent the better part of over 20 years to teach myself through watching videos and practice, however I don't have steady hands and frankly have just been unable to make any vision come to life. Which brings me to the conflict here. This would have been a dream of mine to be able to get frames created and flesh out the story in my head. To make this video professionally, it would be an absurd amount of money. A small studio by itself would be over a million dollars just to start up. To hire a studio would be most likely over ~200k. I would love to make this story I have in my head through legit and traditional means, I would love to start a Kickstarter to get funding and hire artists, voice actors, and a production team. However I know that would most likely be an impossible reach that would further fuel the hate. Regardless, I understand the worry this causes. The fear this produces. However, we can't just ignore where AI is currently at and just tell people not to use it or even worse threaten people who use it. People will always use the shiny new toy in front of them, so the real question is how do we either make it work for us and work along side it, or how do we ACTUALLY implement a method to protect art rather than tell people to not use it. Respect your artists. Review their ToS and don't upload their hard work to a model without permission.

LUͦʷCͦk

18,417 Aufrufe • vor 10 Monaten