Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

This is the coolest Diffusion work I've seen in a while! It generates Visual Anagrams, a type of optical illusion where an image looks like one thing, but changes appearance when transformed. It works with any orthogonal transformation matrices, which luckily include rotation, permutation (jigsaw puzzles), and color negation....

DrJimFan's profile picture

Jim Fan

516,893 subscribers

188,968 Aufrufe • vor 2 Jahren •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation
0:19

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation

AK

41,746 Aufrufe • vor 3 Jahren

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.
0:18

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

MrNeRF

17,052 Aufrufe • vor 1 Jahr

Create a 3D model from a single image, set of images or a text prompt in < 1 minute 😮‍💨 This new AI paper called CAT3D shows us that it’ll keep getting easier to produce 3D models from 2D images — whether it’s a sparser real world 3D scan (a few photos instead of hundreds) or your favorite 2D image generator like Midjourney (just an image). How does this magic work? “This architecture is similar to video diffusion models, but with camera pose embeddings for each image instead of time embeddings. The generated views are passed into a robust 3D reconstruction pipeline to create the 3D representation (Zip-NeRF or 3DGS)”
0:18

Create a 3D model from a single image, set of images or a text prompt in < 1 minute 😮‍💨 This new AI paper called CAT3D shows us that it’ll keep getting easier to produce 3D models from 2D images — whether it’s a sparser real world 3D scan (a few photos instead of hundreds) or your favorite 2D image generator like Midjourney (just an image). How does this magic work? “This architecture is similar to video diffusion models, but with camera pose embeddings for each image instead of time embeddings. The generated views are passed into a robust 3D reconstruction pipeline to create the 3D representation (Zip-NeRF or 3DGS)”

Bilawal Sidhu

92,792 Aufrufe • vor 2 Jahren

From product image to video with just one tool - Dzine As you may have noticed, this is one of my favorite tools. It is also very underrated, as probably 50% of my tutorials include some workflow. I was testing the new image-to-video option today, and I love it. Step - by step guide in comments 🔽 I can do 95% of a workflow without switching between apps. Image generation, Image to image with style reference, background removal, background generation, and 2 frames image to video. The only other app I have been using for this video is CapCut so that I can stitch it together. Step by step 🔽
0:11

From product image to video with just one tool - Dzine As you may have noticed, this is one of my favorite tools. It is also very underrated, as probably 50% of my tutorials include some workflow. I was testing the new image-to-video option today, and I love it. Step - by step guide in comments 🔽 I can do 95% of a workflow without switching between apps. Image generation, Image to image with style reference, background removal, background generation, and 2 frames image to video. The only other app I have been using for this video is CapCut so that I can stitch it together. Step by step 🔽

Teodora P L

28,523 Aufrufe • vor 1 Jahr

✨ I've brought back from the dead It was my first visual AI project in 2022, and it's this project that generated random ArchDaily-style architecture designs that made me realize AI image models could do interior design That led me to make which then led me to finetune my first interior design model which then for fun I uploaded my own photos too, which led me to make and then pivoted that in to So this project has a special place for me It was still alive but wasn't generating new designs anymore because it ran on Stable Diffusion 1.5 which was outdated and everything stopped working about a year ago I've now migrated it to its own Hetzner VPS now, which means I can run Claude Code on the server with it, and cleaned it up and upgraded it to the latest AI image models (including Nano Banana Pro) It now generates about 12 new designs every day again, and you can up or downvote the ones you like or don't like!
0:13

✨ I've brought back from the dead It was my first visual AI project in 2022, and it's this project that generated random ArchDaily-style architecture designs that made me realize AI image models could do interior design That led me to make which then led me to finetune my first interior design model which then for fun I uploaded my own photos too, which led me to make and then pivoted that in to So this project has a special place for me It was still alive but wasn't generating new designs anymore because it ran on Stable Diffusion 1.5 which was outdated and everything stopped working about a year ago I've now migrated it to its own Hetzner VPS now, which means I can run Claude Code on the server with it, and cleaned it up and upgraded it to the latest AI image models (including Nano Banana Pro) It now generates about 12 new designs every day again, and you can up or downvote the ones you like or don't like!

@levelsio

204,387 Aufrufe • vor 4 Monaten

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to *potentially* giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.
0:13

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to *potentially* giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.

nathan (in sf)

40,440 Aufrufe • vor 6 Monaten

Midjourney sref + Sora 2 Pro is the sauce. With one Midjourney style image, you can give a specific style for your entire project. I created two different 12-second clips and edited them together. Some details aren’t fully consistent, like the iPod or AirPods because the clips were made separately from a single image (Character in a specific style). It could be fixed in post-production, but that would take more time, and this was more of an experimental test. It would be great to add the actual product image with the current one to maintain product consistency. I feel like if there were a way to add 2–4 images into this workflow, it could open up a lot more possibilities and consistency. With an API, it could be possible. Or let’s see what Veo 3.1 has to offer.
0:16

Midjourney sref + Sora 2 Pro is the sauce. With one Midjourney style image, you can give a specific style for your entire project. I created two different 12-second clips and edited them together. Some details aren’t fully consistent, like the iPod or AirPods because the clips were made separately from a single image (Character in a specific style). It could be fixed in post-production, but that would take more time, and this was more of an experimental test. It would be great to add the actual product image with the current one to maintain product consistency. I feel like if there were a way to add 2–4 images into this workflow, it could open up a lot more possibilities and consistency. With an API, it could be possible. Or let’s see what Veo 3.1 has to offer.

Allar Haltsonen

10,141 Aufrufe • vor 9 Monaten

Video diffusion models have strong implicit representations of 3D shape, material, and lighting, but controlling them with language is cumbersome, and control is critical for artists and animators. GenLit connects these implicit representations with a continuous 5D control signal describing the direction and intensity of a point light source. This enables single-image near-field relighting of an image using a video diffusion model. We use a ControlNet-like approach and show that, with a small amount of synthetic data, GenLit generalizes to complex real-world images. Given a single image and the 5D lighting signal, GenLit creates a video of a moving light source that is inside the scene. It moves around and behind scene objects, producing effects such as shading, cast shadows, secularities, and interreflections with a realism that is hard to obtain with traditional inverse rendering methods. GenLit shows that it is possible to get continuous control over implicit physical processes within a video model. I think this is just the beginning and promises to make such models much more practical for creators. Shrisha Bharadwaj will present today at SIGGRAPH Asia Room: S423/S424, Level 4 @ 13:50 on 15 of Dec.
0:13

Video diffusion models have strong implicit representations of 3D shape, material, and lighting, but controlling them with language is cumbersome, and control is critical for artists and animators. GenLit connects these implicit representations with a continuous 5D control signal describing the direction and intensity of a point light source. This enables single-image near-field relighting of an image using a video diffusion model. We use a ControlNet-like approach and show that, with a small amount of synthetic data, GenLit generalizes to complex real-world images. Given a single image and the 5D lighting signal, GenLit creates a video of a moving light source that is inside the scene. It moves around and behind scene objects, producing effects such as shading, cast shadows, secularities, and interreflections with a realism that is hard to obtain with traditional inverse rendering methods. GenLit shows that it is possible to get continuous control over implicit physical processes within a video model. I think this is just the beginning and promises to make such models much more practical for creators. Shrisha Bharadwaj will present today at SIGGRAPH Asia Room: S423/S424, Level 4 @ 13:50 on 15 of Dec.

Michael Black

22,182 Aufrufe • vor 7 Monaten

✨ Made a new mini feature on Photo AI: [ Grab from 3d model ] So the problem is we're at that stage in time (typical for AI) where image-to-3d models are not good enough but are fun to play with, but we know they'll be good enough in 1-2 years With [ Make 3d model ] you already can turn any Photo AI pic into a 3d model but it still looks hyper clunky and deformed, but it works! One cool idea I had to make that more useful and made now: Let people make a 3d model then change the view of the it with the 3d viewer, then press [ o ] and it grabs a frame of the 3d That image you can then [ Remix ] (img2img), and it becomes a real photo again and that in turn you can then turn into a video again with [ Make video ] So that essentially gives you a fully freeform camera position control to take photos with One thing I need to fix is the background/skybox, I kinda need to take the original photo and remove the person and just get the background for the 3d model viewer, in this case it should be white, but it's a start!
0:51

✨ Made a new mini feature on Photo AI: [ Grab from 3d model ] So the problem is we're at that stage in time (typical for AI) where image-to-3d models are not good enough but are fun to play with, but we know they'll be good enough in 1-2 years With [ Make 3d model ] you already can turn any Photo AI pic into a 3d model but it still looks hyper clunky and deformed, but it works! One cool idea I had to make that more useful and made now: Let people make a 3d model then change the view of the it with the 3d viewer, then press [ o ] and it grabs a frame of the 3d That image you can then [ Remix ] (img2img), and it becomes a real photo again and that in turn you can then turn into a video again with [ Make video ] So that essentially gives you a fully freeform camera position control to take photos with One thing I need to fix is the background/skybox, I kinda need to take the original photo and remove the person and just get the background for the 3d model viewer, in this case it should be white, but it's a start!

@levelsio

119,210 Aufrufe • vor 1 Jahr

Step 1 : input your char reference, poster and your product into GPT Image 2 and then put the prompt : Use the woman on image 1 as the main subject. create a vertical poster ad inspired by the reference poster style. show the image 1 woman holding the kimchi jar with her left hand, while the right hand eating kimchi to her mouth. with framing wide lens, low angle from the reference poster. This step is where you lock the visual direction. Step 2 : take the generated image into Seedance 2.0 and convert it into motion. Set it to 1080p if your design includes a lot of typography, this helps preserve text clarity. You can also strengthen your prompt by adding keywords like: “dynamic motion design commercial advertisement” to push the result closer to a polished ad style. Important note : If your design contains heavy typography, expect some inconsistencies in text rendering. The best approach is to generate multiple variations and select the cleanest result. For this sample, I generated it 4 times before landing on the final version.
0:15

Step 1 : input your char reference, poster and your product into GPT Image 2 and then put the prompt : Use the woman on image 1 as the main subject. create a vertical poster ad inspired by the reference poster style. show the image 1 woman holding the kimchi jar with her left hand, while the right hand eating kimchi to her mouth. with framing wide lens, low angle from the reference poster. This step is where you lock the visual direction. Step 2 : take the generated image into Seedance 2.0 and convert it into motion. Set it to 1080p if your design includes a lot of typography, this helps preserve text clarity. You can also strengthen your prompt by adding keywords like: “dynamic motion design commercial advertisement” to push the result closer to a polished ad style. Important note : If your design contains heavy typography, expect some inconsistencies in text rendering. The best approach is to generate multiple variations and select the cleanest result. For this sample, I generated it 4 times before landing on the final version.

DStudioproject

11,413 Aufrufe • vor 3 Monaten

Google dropped a new AI paper called LUMIERE. It's remarkably flexible, supporting video inpainting, image-to-video, AND stylized video generation tasks. Say hello to “space-time diffusion” for video generation! Now what the heck does that mean exactly?! 🌐⏳ → TL;DR it utilizes a “Space-Time UNet” architecture that generates the full duration of the video in one pass, rather than generating distant keyframes and interpolating between them like prior works. Because the computation is done in this “compressed space-time representation” to generate the full clip at once, it's far more temporally consistent. → Another benefit of generating the full video at once is that you can “direct” the video generation, making it easier to hand off to other models/tasks without having to stitch together partial solutions. You can condition generations on additional inputs, meaning you get the full stack of AI video capabilities – from video inpainting to image-to-video and beyond. → New SOTA for AI video generation? User study results in the paper suggest human evaluators preferred Lumiere over Runway Gen-2, Pika Labs, and Stable Video Diffusion in terms of quality, text alignment AND motion. But as always, we need to get hands-on with this tech when Google *actually* decides to ship it. → Could this end up inside YouTube? Y’all know i’m obsessed with blending reality and imagination – so it’s the video inpainting tech I'm most excited about. I really hope this model finds its way into YouTube's Generative AI efforts, and based on their prior announcements and the list of acknowledgments in the paper I think it might! 🤞🏽 Links: 🔗Paper: 🔗Project:
0:13

Google dropped a new AI paper called LUMIERE. It's remarkably flexible, supporting video inpainting, image-to-video, AND stylized video generation tasks. Say hello to “space-time diffusion” for video generation! Now what the heck does that mean exactly?! 🌐⏳ → TL;DR it utilizes a “Space-Time UNet” architecture that generates the full duration of the video in one pass, rather than generating distant keyframes and interpolating between them like prior works. Because the computation is done in this “compressed space-time representation” to generate the full clip at once, it's far more temporally consistent. → Another benefit of generating the full video at once is that you can “direct” the video generation, making it easier to hand off to other models/tasks without having to stitch together partial solutions. You can condition generations on additional inputs, meaning you get the full stack of AI video capabilities – from video inpainting to image-to-video and beyond. → New SOTA for AI video generation? User study results in the paper suggest human evaluators preferred Lumiere over Runway Gen-2, Pika Labs, and Stable Video Diffusion in terms of quality, text alignment AND motion. But as always, we need to get hands-on with this tech when Google *actually* decides to ship it. → Could this end up inside YouTube? Y’all know i’m obsessed with blending reality and imagination – so it’s the video inpainting tech I'm most excited about. I really hope this model finds its way into YouTube's Generative AI efforts, and based on their prior announcements and the list of acknowledgments in the paper I think it might! 🤞🏽 Links: 🔗Paper: 🔗Project:

Bilawal Sidhu

44,822 Aufrufe • vor 2 Jahren

this effect is all over tiktok right now and nobody's explaining how to actually do it properly... the 3d balloon character thing. where someone turns into a shiny inflatable version of themselves that still moves and talks. looks pretty smooth in feeds. the workflow is stupid simple once you see it. step 1: take any photo. drop it into an image gen tool (nano banana pro). prompt it with something like "make the person in the photo a plastic blow up balloon character with a shiny surface. keep the face details as 3d balloon details including the person in the background. don't change background" that's it for the image. don't overcomplicate the prompt. shorter = more consistent results. (learned this after wasting like 2 hours trying to get "perfect" prompts that kept giving me garbage) step 2: take that balloon image + your original video and drop both into kling motion control. prompt: "turn the motion and detailed mouth movement of the video to the setting of the image" that's literally it. kling maps the motion from the real video onto the balloon character. mouth moves. head turns. expressions transfer. the whole thing renders in a few minutes. the result looks like a $500 custom animation and costs you maybe $0.30 in kling credits. people are getting 500k+ views with these because the scroll-stop factor is insane. nobody expects to see a shiny inflatable version of someone giving a real speech or doing a product review. the play here is obvious btw. run this for client content (mix with the hook and real body, check the results yourself) or use it on your own faceless channels as a hook pattern before the algo catches up...
0:17

this effect is all over tiktok right now and nobody's explaining how to actually do it properly... the 3d balloon character thing. where someone turns into a shiny inflatable version of themselves that still moves and talks. looks pretty smooth in feeds. the workflow is stupid simple once you see it. step 1: take any photo. drop it into an image gen tool (nano banana pro). prompt it with something like "make the person in the photo a plastic blow up balloon character with a shiny surface. keep the face details as 3d balloon details including the person in the background. don't change background" that's it for the image. don't overcomplicate the prompt. shorter = more consistent results. (learned this after wasting like 2 hours trying to get "perfect" prompts that kept giving me garbage) step 2: take that balloon image + your original video and drop both into kling motion control. prompt: "turn the motion and detailed mouth movement of the video to the setting of the image" that's literally it. kling maps the motion from the real video onto the balloon character. mouth moves. head turns. expressions transfer. the whole thing renders in a few minutes. the result looks like a $500 custom animation and costs you maybe $0.30 in kling credits. people are getting 500k+ views with these because the scroll-stop factor is insane. nobody expects to see a shiny inflatable version of someone giving a real speech or doing a product review. the play here is obvious btw. run this for client content (mix with the hook and real body, check the results yourself) or use it on your own faceless channels as a hook pattern before the algo catches up...

KNOX

25,773 Aufrufe • vor 5 Monaten

Everyone's sleeping on image-to-3D AI models. They can make your app look incredibly unique, with just a little effort. Here's how. This is my calorie tracker, built in a week with nothing but prompting. Just Claude Code + a couple APIs. The visuals are all AI-generated. I'll be sharing the full workflow + all the crazy technical stuff Claude and I did to make this work, so nobody has to struggle through it like me. Deep dive coming soon! Till then, this is the high-level idea: 1. Get a clean image of the food (or whatever your asset is) - In my app, the user describes foods via text, or attaches images (or both) - If text, an LLM extracts the food description and formats it into a specific prompt I tuned for this design, and we generate an image using Z-Image Turbo through fal - If image, we do the same thing but with FLUX.2 [dev] to edit the user image into our reference design - Originally, both used Google Nano Banana, but switching to open models cut costs and latency a ton 2. Gaussian splatting (2D image → 3D model) - I tried various 2D-to-3D options on fal and ended up with TripoSplat as my preferred balance of speed, cost, latency; this turns an image into a 3D model that looks super high quality (link below) - The app displays the 2D image while our backend generates the 3D splat - We "groom" the splat to reduce size and load time by culling low-opacity/scale points 3. Render efficiently on device Originally, it looked great but ran at 10 FPS. Getting to 120 FPS was a crazy journey. TL;DR: - SwiftUI had to go; it forced us to render each asset in independent MTKViews, which wasn't workable - Instead, we composite every dish into one full-bleed CAMetalLayer using MetalSplatter (link below) - We had to make some optimizations within MetalSplatter's code too, to reduce the overhead of sorting points per render Then I added some finishing touches like the subtle rotation and parallax as they move around. I think it turned out pretty cool :) Overall, this took some effort, but we still got it done in less than a day. Hopefully your agent can follow in the footsteps of mine and do it much faster. Keep an eye out for the bigger writeup, which'll give your agent everything it needs. If you have any questions, drop em below!
0:14

Everyone's sleeping on image-to-3D AI models. They can make your app look incredibly unique, with just a little effort. Here's how. This is my calorie tracker, built in a week with nothing but prompting. Just Claude Code + a couple APIs. The visuals are all AI-generated. I'll be sharing the full workflow + all the crazy technical stuff Claude and I did to make this work, so nobody has to struggle through it like me. Deep dive coming soon! Till then, this is the high-level idea: 1. Get a clean image of the food (or whatever your asset is) - In my app, the user describes foods via text, or attaches images (or both) - If text, an LLM extracts the food description and formats it into a specific prompt I tuned for this design, and we generate an image using Z-Image Turbo through fal - If image, we do the same thing but with FLUX.2 [dev] to edit the user image into our reference design - Originally, both used Google Nano Banana, but switching to open models cut costs and latency a ton 2. Gaussian splatting (2D image → 3D model) - I tried various 2D-to-3D options on fal and ended up with TripoSplat as my preferred balance of speed, cost, latency; this turns an image into a 3D model that looks super high quality (link below) - The app displays the 2D image while our backend generates the 3D splat - We "groom" the splat to reduce size and load time by culling low-opacity/scale points 3. Render efficiently on device Originally, it looked great but ran at 10 FPS. Getting to 120 FPS was a crazy journey. TL;DR: - SwiftUI had to go; it forced us to render each asset in independent MTKViews, which wasn't workable - Instead, we composite every dish into one full-bleed CAMetalLayer using MetalSplatter (link below) - We had to make some optimizations within MetalSplatter's code too, to reduce the overhead of sorting points per render Then I added some finishing touches like the subtle rotation and parallax as they move around. I think it turned out pretty cool :) Overall, this took some effort, but we still got it done in less than a day. Hopefully your agent can follow in the footsteps of mine and do it much faster. Keep an eye out for the bigger writeup, which'll give your agent everything it needs. If you have any questions, drop em below!

Anshu

19,931 Aufrufe • vor 1 Monat

🧪 My GEN-3 Prompting Process I get a lot of questions on how I find my prompts when using Image-2-Video in Runway. Here is a quick breakdown of the general thought process. If you have any further questions, let's chat in the comments below. 1️⃣ I always start by doing 3x generations without any prompts and additional settings. 2️⃣ I analyze those 3x generations and identify patterns. What did the model always do well, where did it fail. 3️⃣ I then use prompting and the different controllability features to eliminate where the model struggled on its own. --- General Tips --- *️⃣ There are some tokens which work universally. "Muted colors, low contrast" are great to preserve the colors of the original input image. "Static camera, natural movement" works fantastically to get cinematic shots. *️⃣ My I2V prompts are on the shorter side. It's usually a sentence describing the scene and then individual modifiers like the ones mentioned above to fix certain camera/lighting/movement artifacts. *️⃣ Start small and prompt engineer in steps. This is very much an iterative process which rewards you for understanding model behavior and knowing how to craft a visual architecture with words. --- Disclaimer --- Please note that this approach is more suitable for a professional workflow. Therefore, I recommend it for users on the unlimited plan who don't need to worry about credits.
0:10

🧪 My GEN-3 Prompting Process I get a lot of questions on how I find my prompts when using Image-2-Video in Runway. Here is a quick breakdown of the general thought process. If you have any further questions, let's chat in the comments below. 1️⃣ I always start by doing 3x generations without any prompts and additional settings. 2️⃣ I analyze those 3x generations and identify patterns. What did the model always do well, where did it fail. 3️⃣ I then use prompting and the different controllability features to eliminate where the model struggled on its own. --- General Tips --- *️⃣ There are some tokens which work universally. "Muted colors, low contrast" are great to preserve the colors of the original input image. "Static camera, natural movement" works fantastically to get cinematic shots. *️⃣ My I2V prompts are on the shorter side. It's usually a sentence describing the scene and then individual modifiers like the ones mentioned above to fix certain camera/lighting/movement artifacts. *️⃣ Start small and prompt engineer in steps. This is very much an iterative process which rewards you for understanding model behavior and knowing how to craft a visual architecture with words. --- Disclaimer --- Please note that this approach is more suitable for a professional workflow. Therefore, I recommend it for users on the unlimited plan who don't need to worry about credits.

Nicolas Neubert

46,240 Aufrufe • vor 1 Jahr

A preview of what's next, visualized with Rerun and PlayCanvas supersplat ✨ (Also, feel free to send me a DM 📩; I’ll be in San Francisco from July 21–29, and I'm looking to meet like-minded folks!) I'm convinced that Gaussian Splats will be an integral part of any data engine as an underlying representation. So I've started putting together a repo that: 1. Given a single image, perform image outpainting 🖼️🖌️ 2. Estimate a monocular depth map on the outpainted image 📏 3. Train a Gaussian Splat initialized from the monocular depth 🎓✨ 4. Warp to new views, perform inpainting on the missing masks -> Train new splat 🔄🎨 This is going to be integrated into exo-egoforge, but I wanted to start with the simple single-image version before moving to a multi-video implementation There's some weirdness in the final rerun visualization, but the trained splat looks great 🎉! This is all based on the very cool VistaDream paper ( .github.io/) More on this next week!
0:18

A preview of what's next, visualized with Rerun and PlayCanvas supersplat ✨ (Also, feel free to send me a DM 📩; I’ll be in San Francisco from July 21–29, and I'm looking to meet like-minded folks!) I'm convinced that Gaussian Splats will be an integral part of any data engine as an underlying representation. So I've started putting together a repo that: 1. Given a single image, perform image outpainting 🖼️🖌️ 2. Estimate a monocular depth map on the outpainted image 📏 3. Train a Gaussian Splat initialized from the monocular depth 🎓✨ 4. Warp to new views, perform inpainting on the missing masks -> Train new splat 🔄🎨 This is going to be integrated into exo-egoforge, but I wanted to start with the simple single-image version before moving to a multi-video implementation There's some weirdness in the final rerun visualization, but the trained splat looks great 🎉! This is all based on the very cool VistaDream paper ( .github.io/) More on this next week!

Pablo Vela

26,036 Aufrufe • vor 1 Jahr

✨ Grok's new Imagine video model also comes with an Edit model We know edit models for images, you submit an image, write a prompt what to change, but this is the first time I've seen a proper edit model for video And it kinda works, not great yet though but it does something Here I had to remove the old name "Nomad List" in the video for my site First it said "Go nomad -> Nomad List", so I prompted it "remove the text Nomad List. do not change anything else", it didn't remove it but it replaced it with just "Go nomad" again, okay good enough Useful because otherwise I'd have to scour my backups for the original video in Final Cut Pro, and this is faster One thing you see is it changes the pattern on the door also, but that's okay for now if I fade it in
0:17

✨ Grok's new Imagine video model also comes with an Edit model We know edit models for images, you submit an image, write a prompt what to change, but this is the first time I've seen a proper edit model for video And it kinda works, not great yet though but it does something Here I had to remove the old name "Nomad List" in the video for my site First it said "Go nomad -> Nomad List", so I prompted it "remove the text Nomad List. do not change anything else", it didn't remove it but it replaced it with just "Go nomad" again, okay good enough Useful because otherwise I'd have to scour my backups for the original video in Final Cut Pro, and this is faster One thing you see is it changes the pattern on the door also, but that's okay for now if I fade it in

@levelsio

73,183 Aufrufe • vor 5 Monaten

I HAVE A SCHIZO THEORY! I noticed something in the combat footage. Right after Edward assassinates an enemy with the hidden blade, there's this really brief moment where his stance changes into the hidden blade combat stance, then it snaps straight back to normal. It's like half a second. Obviously, it's most likely because the game's still a work in progress, but it got me thinking… what if there's actually a hopium chance that hidden blade combat is being implemented? The stance he switches into does look like it's the hidden blade combat stance though and not unarmed stance. See the image.
0:15

I HAVE A SCHIZO THEORY! I noticed something in the combat footage. Right after Edward assassinates an enemy with the hidden blade, there's this really brief moment where his stance changes into the hidden blade combat stance, then it snaps straight back to normal. It's like half a second. Obviously, it's most likely because the game's still a work in progress, but it got me thinking… what if there's actually a hopium chance that hidden blade combat is being implemented? The stance he switches into does look like it's the hidden blade combat stance though and not unarmed stance. See the image.

The Hidden One

121,389 Aufrufe • vor 2 Monaten

Unpopular opinion: Most agent evals are theatre. You run them once before the deployment. It'll take 800ms+ as another LLM would be judging your LLM. Most annoying part - no one tells where in the chain things went wrong. I wasted a lot of time in this loop. And then I came across Future AGI bringing 5 different tools under one umbrella, best part - the platform is completely open source. They open sourced their entire platform and the eval layer is noticeably different. It is multimodal - works on everything text, image, audio, pdf. Not an LLM-as-judge adding latency but an agent with memory and tools. The biggest win are learned classifiers trained on actual production failure patterns to run evals at low cost. It also runs across the full reasoning chain, not just the final response. Check out → Try it here →
0:13

Unpopular opinion: Most agent evals are theatre. You run them once before the deployment. It'll take 800ms+ as another LLM would be judging your LLM. Most annoying part - no one tells where in the chain things went wrong. I wasted a lot of time in this loop. And then I came across Future AGI bringing 5 different tools under one umbrella, best part - the platform is completely open source. They open sourced their entire platform and the eval layer is noticeably different. It is multimodal - works on everything text, image, audio, pdf. Not an LLM-as-judge adding latency but an agent with memory and tools. The biggest win are learned classifiers trained on actual production failure patterns to run evals at low cost. It also runs across the full reasoning chain, not just the final response. Check out → Try it here →

Swapna Kumar Panda

50,102 Aufrufe • vor 3 Monaten

gemini omniflash is actually f*cking cracked. you can animate/edit any video with a text prompt. character swaps, object transforms, full environment changes without regenerating/rotoscoping. everyone using AI to to animate and edit videos right now hits the same wall. the clip comes out 90% right and you regenerate from scratch hoping the 10% fixes itself. it never does. the fix is using your video as the input. omniflash edits what's already there instead of rolling the dice again. here's what's in the system: > the two-layer premiere trick: generate the same shot twice (one with background removed), stack them, cut at one frame, instant scene change > character swap with a single reference image (plus the one line you need or the model keeps the original's features) > object transforms that leave the rest of the frame untouched: stone into glowing sphere, candles into flowers > style transfer from an image reference instead of text, way more accurate > why stacking edits in one prompt breaks everything and the exact step order that doesn't > the audio limitation nobody mentions and how to work around it i packaged every prompt, the edit sequence, and the premiere layering setup. RT + reply "OMNI" and i'll send it over.
0:19

gemini omniflash is actually f*cking cracked. you can animate/edit any video with a text prompt. character swaps, object transforms, full environment changes without regenerating/rotoscoping. everyone using AI to to animate and edit videos right now hits the same wall. the clip comes out 90% right and you regenerate from scratch hoping the 10% fixes itself. it never does. the fix is using your video as the input. omniflash edits what's already there instead of rolling the dice again. here's what's in the system: > the two-layer premiere trick: generate the same shot twice (one with background removed), stack them, cut at one frame, instant scene change > character swap with a single reference image (plus the one line you need or the model keeps the original's features) > object transforms that leave the rest of the frame untouched: stone into glowing sphere, candles into flowers > style transfer from an image reference instead of text, way more accurate > why stacking edits in one prompt breaks everything and the exact step order that doesn't > the audio limitation nobody mentions and how to work around it i packaged every prompt, the edit sequence, and the premiere layering setup. RT + reply "OMNI" and i'll send it over.

Sulfur

36,103 Aufrufe • vor 26 Tagen

V TYPE 非: What Does Kim Taehyung's Creative Direction Mean? The announcement of the V TYPE 非 photobook highlighted a detail that defines the significance of this work: the artist actively participated in every stage, “from planning to photography.” But what does this mean in practice? A Creative Director is responsible for conceiving, organizing, and guiding the artistic vision of a project. Their role is not to technically execute all the steps, but to define the “why,” the “how,” and the “what for” of each aesthetic and narrative choice. They act as a true ARCHITECT OF IDEAS, ensuring that concept, image, narrative, and emotion work together. They ensure that the project has soul, identity, and coherence, aligning all the professionals involved — photographers, designers, stylists, costume designers, set designers, and editors — in a single creative direction. IN GENERAL, KIM TAEHYUNG: • Created the central concept • Defined the visual and emotional narrative • Established artistic and symbolic references (art, fashion, cinema, photography, music, architecture, etc.) • Guided the aesthetic language (atmosphere, light, palette, rhythm, textures) • Determined the tone of the project (intimate, experimental, documentary, symbolic) • Established the visual rhythm: order of images, pauses, silences • Guided framing, expressions, poses, and atmospheres • Coordinated and aligned the creative teams • Ensured coherence and identity from beginning to end • Conceived the Photobook as an artistic object: paper, cover, typography, weight, and texture. Taehyung is the guardian of the artistic vision. The result is a cohesive visual discourse, where each image exists for a reason. THE CREATIVE DIRECTOR IN THE PHOTOBOOK EXHIBITION In the exhibition, the creative director expands the photobook into physical space, transforming it into an immersive experience. The purpose is not just to show photos: it's to make the public traverse the concept. HERE, KIM TAEHYUNG ACTS IN: • Curating the exhibited images • Defining the public's path • Building the spatial narrative (beginning, climax, ending) • Dialogue between images, space, light, and silence • Choosing formats, sizes, and layouts • Integrating sensory elements (sound, text, architecture, rhythm) • Ensuring that the exhibition preserves the original concept of the photobook. In a market where many artists merely lend their image, Taehyung reaffirms his role as an author. “TYPE 非” is not just a record of his image, it is the materialization of his artistic sensibility. It is the signature of someone who not only participates in the work — he builds the work. In a photobook and its exhibition, Kim Tarhyung directs the experience, the reading, and the memory that the public will take with them. V TYPE 非 IS COMING!!! #V_TYPE非
1:29

V TYPE 非: What Does Kim Taehyung's Creative Direction Mean? The announcement of the V TYPE 非 photobook highlighted a detail that defines the significance of this work: the artist actively participated in every stage, “from planning to photography.” But what does this mean in practice? A Creative Director is responsible for conceiving, organizing, and guiding the artistic vision of a project. Their role is not to technically execute all the steps, but to define the “why,” the “how,” and the “what for” of each aesthetic and narrative choice. They act as a true ARCHITECT OF IDEAS, ensuring that concept, image, narrative, and emotion work together. They ensure that the project has soul, identity, and coherence, aligning all the professionals involved — photographers, designers, stylists, costume designers, set designers, and editors — in a single creative direction. IN GENERAL, KIM TAEHYUNG: • Created the central concept • Defined the visual and emotional narrative • Established artistic and symbolic references (art, fashion, cinema, photography, music, architecture, etc.) • Guided the aesthetic language (atmosphere, light, palette, rhythm, textures) • Determined the tone of the project (intimate, experimental, documentary, symbolic) • Established the visual rhythm: order of images, pauses, silences • Guided framing, expressions, poses, and atmospheres • Coordinated and aligned the creative teams • Ensured coherence and identity from beginning to end • Conceived the Photobook as an artistic object: paper, cover, typography, weight, and texture. Taehyung is the guardian of the artistic vision. The result is a cohesive visual discourse, where each image exists for a reason. THE CREATIVE DIRECTOR IN THE PHOTOBOOK EXHIBITION In the exhibition, the creative director expands the photobook into physical space, transforming it into an immersive experience. The purpose is not just to show photos: it's to make the public traverse the concept. HERE, KIM TAEHYUNG ACTS IN: • Curating the exhibited images • Defining the public's path • Building the spatial narrative (beginning, climax, ending) • Dialogue between images, space, light, and silence • Choosing formats, sizes, and layouts • Integrating sensory elements (sound, text, architecture, rhythm) • Ensuring that the exhibition preserves the original concept of the photobook. In a market where many artists merely lend their image, Taehyung reaffirms his role as an author. “TYPE 非” is not just a record of his image, it is the materialization of his artistic sensibility. It is the signature of someone who not only participates in the work — he builds the work. In a photobook and its exhibition, Kim Tarhyung directs the experience, the reading, and the memory that the public will take with them. V TYPE 非 IS COMING!!! #V_TYPE非

Music Map ⓥ

28,560 Aufrufe • vor 6 Monaten