This is the coolest Diffusion work I've seen in... a while! It generates Visual Anagrams, a type of optical illusion where an image looks like one thing, but changes appearance when transformed. It works with any orthogonal transformation matrices, which luckily include rotation, permutation (jigsaw puzzles), and color negation. Intuitively, the method first inverts the noise from multiple image transforms (with different text prompts), and then average them. After taking a diffusion step in the averaged noise, the resulting image becomes an anagram that aligns with the texts in different views. It does very little computation, using pre-trained Stable Diffusion. Simple, elegant, and inexpensive technique for non-professionals to create some interesting art! Paper: Website: It's open-source! Authors: Daniel Geng, Inbum Park, Andrew Owens.show more

The Hidden Language of Diffusion Models paper page: tackle... the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulationshow more

DimensionX: Create Any 3D and 4D Scenes from a... Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.show more

Create a 3D model from a single image, set... of images or a text prompt in < 1 minute 😮‍💨 This new AI paper called CAT3D shows us that it’ll keep getting easier to produce 3D models from 2D images — whether it’s a sparser real world 3D scan (a few photos instead of hundreds) or your favorite 2D image generator like Midjourney (just an image). How does this magic work? “This architecture is similar to video diffusion models, but with camera pose embeddings for each image instead of time embeddings. The generated views are passed into a robust 3D reconstruction pipeline to create the 3D representation (Zip-NeRF or 3DGS)”show more

MiniMax H3 Instead of sharing the prompts for each... of these videos, I thought it would be more useful to share how I created that prompts. All of the videos were generated with text-to-video. First, find an image with the kind of scene, composition and mood you want to recreate. I used a few YouTube playlist thumbnails as references but Pinterest is also a great place to find inspiration. You can even use your own old or nostalgic photographs. Then upload the image to ChatGPT and ask it to describe the scene. The description it gives you can essentially become your text-to-video prompt. From there, you can generate completely new scenes with a similar composition, atmosphere and cinematic language. You can of course use the reference image directly with image-to-video or as a first frame. But if the original image isn't yours, I prefer using it only as visual inspiration and recreating the scene through text-to-video. This is the prompt I use with ChatGPT: "Describe the scene in this image in English, focusing primarily on what is happening, the characters, their actions and body language, the setting and the overall atmosphere. Also briefly describe the composition, framing, camera angle, approximate lens choice, lighting, color palette and cinematic aesthetic. Keep it concise and scene-focused rather than overly technical."show more

From product image to video with just one tool... - Dzine As you may have noticed, this is one of my favorite tools. It is also very underrated, as probably 50% of my tutorials include some workflow. I was testing the new image-to-video option today, and I love it. Step - by step guide in comments 🔽 I can do 95% of a workflow without switching between apps. Image generation, Image to image with style reference, background removal, background generation, and 2 frames image to video. The only other app I have been using for this video is CapCut so that I can stitch it together. Step by step 🔽show more

✨ I've brought back from the dead It was... my first visual AI project in 2022, and it's this project that generated random ArchDaily-style architecture designs that made me realize AI image models could do interior design That led me to make which then led me to finetune my first interior design model which then for fun I uploaded my own photos too, which led me to make and then pivoted that in to So this project has a special place for me It was still alive but wasn't generating new designs anymore because it ran on Stable Diffusion 1.5 which was outdated and everything stopped working about a year ago I've now migrated it to its own Hetzner VPS now, which means I can run Claude Code on the server with it, and cleaned it up and upgraded it to the latest AI image models (including Nano Banana Pro) It now generates about 12 new designs every day again, and you can up or downvote the ones you like or don't like!show more

Most recent diffusion language model research (that I’ve seen)... seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to *potentially* giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.show more

Midjourney sref + Sora 2 Pro is the sauce.... With one Midjourney style image, you can give a specific style for your entire project. I created two different 12-second clips and edited them together. Some details aren’t fully consistent, like the iPod or AirPods because the clips were made separately from a single image (Character in a specific style). It could be fixed in post-production, but that would take more time, and this was more of an experimental test. It would be great to add the actual product image with the current one to maintain product consistency. I feel like if there were a way to add 2–4 images into this workflow, it could open up a lot more possibilities and consistency. With an API, it could be possible. Or let’s see what Veo 3.1 has to offer.show more

Video diffusion models have strong implicit representations of 3D... shape, material, and lighting, but controlling them with language is cumbersome, and control is critical for artists and animators. GenLit connects these implicit representations with a continuous 5D control signal describing the direction and intensity of a point light source. This enables single-image near-field relighting of an image using a video diffusion model. We use a ControlNet-like approach and show that, with a small amount of synthetic data, GenLit generalizes to complex real-world images. Given a single image and the 5D lighting signal, GenLit creates a video of a moving light source that is inside the scene. It moves around and behind scene objects, producing effects such as shading, cast shadows, secularities, and interreflections with a realism that is hard to obtain with traditional inverse rendering methods. GenLit shows that it is possible to get continuous control over implicit physical processes within a video model. I think this is just the beginning and promises to make such models much more practical for creators. Shrisha Bharadwaj will present today at SIGGRAPH Asia Room: S423/S424, Level 4 @ 13:50 on 15 of Dec.show more

Google dropped a new AI paper called LUMIERE. It's... remarkably flexible, supporting video inpainting, image-to-video, AND stylized video generation tasks. Say hello to “space-time diffusion” for video generation! Now what the heck does that mean exactly?! 🌐⏳ → TL;DR it utilizes a “Space-Time UNet” architecture that generates the full duration of the video in one pass, rather than generating distant keyframes and interpolating between them like prior works. Because the computation is done in this “compressed space-time representation” to generate the full clip at once, it's far more temporally consistent. → Another benefit of generating the full video at once is that you can “direct” the video generation, making it easier to hand off to other models/tasks without having to stitch together partial solutions. You can condition generations on additional inputs, meaning you get the full stack of AI video capabilities – from video inpainting to image-to-video and beyond. → New SOTA for AI video generation? User study results in the paper suggest human evaluators preferred Lumiere over Runway Gen-2, Pika Labs, and Stable Video Diffusion in terms of quality, text alignment AND motion. But as always, we need to get hands-on with this tech when Google *actually* decides to ship it. → Could this end up inside YouTube? Y’all know i’m obsessed with blending reality and imagination – so it’s the video inpainting tech I'm most excited about. I really hope this model finds its way into YouTube's Generative AI efforts, and based on their prior announcements and the list of acknowledgments in the paper I think it might! 🤞🏽 Links: 🔗Paper: 🔗Project:show more

Step 1 : input your char reference, poster and... your product into GPT Image 2 and then put the prompt : Use the woman on image 1 as the main subject. create a vertical poster ad inspired by the reference poster style. show the image 1 woman holding the kimchi jar with her left hand, while the right hand eating kimchi to her mouth. with framing wide lens, low angle from the reference poster. This step is where you lock the visual direction. Step 2 : take the generated image into Seedance 2.0 and convert it into motion. Set it to 1080p if your design includes a lot of typography, this helps preserve text clarity. You can also strengthen your prompt by adding keywords like: “dynamic motion design commercial advertisement” to push the result closer to a polished ad style. Important note : If your design contains heavy typography, expect some inconsistencies in text rendering. The best approach is to generate multiple variations and select the cleanest result. For this sample, I generated it 4 times before landing on the final version.show more

this effect is all over tiktok right now and... nobody's explaining how to actually do it properly... the 3d balloon character thing. where someone turns into a shiny inflatable version of themselves that still moves and talks. looks pretty smooth in feeds. the workflow is stupid simple once you see it. step 1: take any photo. drop it into an image gen tool (nano banana pro). prompt it with something like "make the person in the photo a plastic blow up balloon character with a shiny surface. keep the face details as 3d balloon details including the person in the background. don't change background" that's it for the image. don't overcomplicate the prompt. shorter = more consistent results. (learned this after wasting like 2 hours trying to get "perfect" prompts that kept giving me garbage) step 2: take that balloon image + your original video and drop both into kling motion control. prompt: "turn the motion and detailed mouth movement of the video to the setting of the image" that's literally it. kling maps the motion from the real video onto the balloon character. mouth moves. head turns. expressions transfer. the whole thing renders in a few minutes. the result looks like a $500 custom animation and costs you maybe $0.30 in kling credits. people are getting 500k+ views with these because the scroll-stop factor is insane. nobody expects to see a shiny inflatable version of someone giving a real speech or doing a product review. the play here is obvious btw. run this for client content (mix with the hook and real body, check the results yourself) or use it on your own faceless channels as a hook pattern before the algo catches up...show more

Everyone's sleeping on image-to-3D AI models. They can make... your app look incredibly unique, with just a little effort. Here's how. This is my calorie tracker, built in a week with nothing but prompting. Just Claude Code + a couple APIs. The visuals are all AI-generated. I'll be sharing the full workflow + all the crazy technical stuff Claude and I did to make this work, so nobody has to struggle through it like me. Deep dive coming soon! Till then, this is the high-level idea: 1. Get a clean image of the food (or whatever your asset is) - In my app, the user describes foods via text, or attaches images (or both) - If text, an LLM extracts the food description and formats it into a specific prompt I tuned for this design, and we generate an image using Z-Image Turbo through fal - If image, we do the same thing but with FLUX.2 [dev] to edit the user image into our reference design - Originally, both used Google Nano Banana, but switching to open models cut costs and latency a ton 2. Gaussian splatting (2D image → 3D model) - I tried various 2D-to-3D options on fal and ended up with TripoSplat as my preferred balance of speed, cost, latency; this turns an image into a 3D model that looks super high quality (link below) - The app displays the 2D image while our backend generates the 3D splat - We "groom" the splat to reduce size and load time by culling low-opacity/scale points 3. Render efficiently on device Originally, it looked great but ran at 10 FPS. Getting to 120 FPS was a crazy journey. TL;DR: - SwiftUI had to go; it forced us to render each asset in independent MTKViews, which wasn't workable - Instead, we composite every dish into one full-bleed CAMetalLayer using MetalSplatter (link below) - We had to make some optimizations within MetalSplatter's code too, to reduce the overhead of sorting points per render Then I added some finishing touches like the subtle rotation and parallax as they move around. I think it turned out pretty cool :) Overall, this took some effort, but we still got it done in less than a day. Hopefully your agent can follow in the footsteps of mine and do it much faster. Keep an eye out for the bigger writeup, which'll give your agent everything it needs. If you have any questions, drop em below!show more

omni motion control is f*cking cracked the left clip... is the original. the right is 100% ai, same video, one prompt i grabbed a weight loss transformation off tiktok, gave omni a single image of a totally different girl, and it rebuilt the whole clip around her. she copies the original move for move, down to the dumbbells and the timing. and she doesnt exist and i barely lifted a finger. this was the entire prompt, word for word: "replace the person in the video attached with the person attached in the image, make sure the weight loss transformation is exactly the same from fat to fit" gpt image 2 made the character and omni did the rest. thats it brands pay creators to reshoot the same winning ad with different people just to split test. now you film it once and swap the face for cents. one video turns into 50 i wrote up the full setup: > the prompt method that transfers motion without breaking > how to find reference videos that work > how to batch this out at scale comment "MOTION" and ill send it overshow more

🧪 My GEN-3 Prompting Process I get a lot... of questions on how I find my prompts when using Image-2-Video in Runway. Here is a quick breakdown of the general thought process. If you have any further questions, let's chat in the comments below. 1️⃣ I always start by doing 3x generations without any prompts and additional settings. 2️⃣ I analyze those 3x generations and identify patterns. What did the model always do well, where did it fail. 3️⃣ I then use prompting and the different controllability features to eliminate where the model struggled on its own. --- General Tips --- *️⃣ There are some tokens which work universally. "Muted colors, low contrast" are great to preserve the colors of the original input image. "Static camera, natural movement" works fantastically to get cinematic shots. *️⃣ My I2V prompts are on the shorter side. It's usually a sentence describing the scene and then individual modifiers like the ones mentioned above to fix certain camera/lighting/movement artifacts. *️⃣ Start small and prompt engineer in steps. This is very much an iterative process which rewards you for understanding model behavior and knowing how to craft a visual architecture with words. --- Disclaimer --- Please note that this approach is more suitable for a professional workflow. Therefore, I recommend it for users on the unlimited plan who don't need to worry about credits.show more

A preview of what's next, visualized with Rerun and... PlayCanvas supersplat ✨ (Also, feel free to send me a DM 📩; I’ll be in San Francisco from July 21–29, and I'm looking to meet like-minded folks!) I'm convinced that Gaussian Splats will be an integral part of any data engine as an underlying representation. So I've started putting together a repo that: 1. Given a single image, perform image outpainting 🖼️🖌️ 2. Estimate a monocular depth map on the outpainted image 📏 3. Train a Gaussian Splat initialized from the monocular depth 🎓✨ 4. Warp to new views, perform inpainting on the missing masks -> Train new splat 🔄🎨 This is going to be integrated into exo-egoforge, but I wanted to start with the simple single-image version before moving to a multi-video implementation There's some weirdness in the final rerun visualization, but the trained splat looks great 🎉! This is all based on the very cool VistaDream paper ( .github.io/) More on this next week!show more

✨ Grok's new Imagine video model also comes with... an Edit model We know edit models for images, you submit an image, write a prompt what to change, but this is the first time I've seen a proper edit model for video And it kinda works, not great yet though but it does something Here I had to remove the old name "Nomad List" in the video for my site First it said "Go nomad -> Nomad List", so I prompted it "remove the text Nomad List. do not change anything else", it didn't remove it but it replaced it with just "Go nomad" again, okay good enough Useful because otherwise I'd have to scour my backups for the original video in Final Cut Pro, and this is faster One thing you see is it changes the pattern on the door also, but that's okay for now if I fade it inshow more

I HAVE A SCHIZO THEORY! I noticed something in... the combat footage. Right after Edward assassinates an enemy with the hidden blade, there's this really brief moment where his stance changes into the hidden blade combat stance, then it snaps straight back to normal. It's like half a second. Obviously, it's most likely because the game's still a work in progress, but it got me thinking… what if there's actually a hopium chance that hidden blade combat is being implemented? The stance he switches into does look like it's the hidden blade combat stance though and not unarmed stance. See the image.show more

Unpopular opinion: Most agent evals are theatre. You run... them once before the deployment. It'll take 800ms+ as another LLM would be judging your LLM. Most annoying part - no one tells where in the chain things went wrong. I wasted a lot of time in this loop. And then I came across Future AGI bringing 5 different tools under one umbrella, best part - the platform is completely open source. They open sourced their entire platform and the eval layer is noticeably different. It is multimodal - works on everything text, image, audio, pdf. Not an LLM-as-judge adding latency but an agent with memory and tools. The biggest win are learned classifiers trained on actual production failure patterns to run evals at low cost. It also runs across the full reasoning chain, not just the final response. Check out → Try it here →show more