A preview of what's next, visualized with Rerun and... PlayCanvas supersplat ✨ (Also, feel free to send me a DM 📩; I’ll be in San Francisco from July 21–29, and I'm looking to meet like-minded folks!) I'm convinced that Gaussian Splats will be an integral part of any data engine as an underlying representation. So I've started putting together a repo that: 1. Given a single image, perform image outpainting 🖼️🖌️ 2. Estimate a monocular depth map on the outpainted image 📏 3. Train a Gaussian Splat initialized from the monocular depth 🎓✨ 4. Warp to new views, perform inpainting on the missing masks -> Train new splat 🔄🎨 This is going to be integrated into exo-egoforge, but I wanted to start with the simple single-image version before moving to a multi-video implementation There's some weirdness in the final rerun visualization, but the trained splat looks great 🎉! This is all based on the very cool VistaDream paper ( .github.io/) More on this next week!show more

Some updates on the multiview vistadream pipeline with Rerun!... Rerun came in extremely useful here, as being able to visualize depths at each stage of the pipeline allowed me to debug some nasty bugs. Since the last time, I was only working with a single image input. I've added in VGGT as my multiview pose + depth estimator. It works REALLY well for getting camera poses, but the depths are not that great. To try and fix that, I estimated depth maps from MoGeV2 for each of the views, and scale+shift aligned them so that they would match up to the confident sections of VGGT's depth predictions. You can see in the video just how much sharper the visualized 2d depth maps are! The biggest issue continues to be the multiview consistency 🫠 That's up next, along with actually training the Gaussian splat. Lots of work went into actually understanding inputs+outputs for VGGT. I had some funky bugs where the confidence values would all collapse to true I'm also really excited for this pipeline to use Difix3D+ Nvidia instead of Flux Inpainting, it seems like a better suited for a multiview pipeline.show more

I've been working a lot with SAM3 and the... Momentum Human Rig (MHR). I finally integrated it into the data I'm working with Rerun. The progression I've taken looks as follows SAM3 + SAM3D-body on 1. a single image 2. a set of multiple images 3. a single video 4. A multiview video capture I took inspiration from the SAM3D-body paper and built a multiview fitting optimization pipeline. This pipeline involves using the 2D keypoints from the single-view pipeline, triangulating them, and employing an L1 loss between the 2D/3D keypoints. The temporal stability isn't great, so that's the next portion I'm going to focus on. One really frustrating thing about SAM3D-body is the lack of per-joint confidence values. It makes it harder to deal with occlusions. I'm probably going to need to use a separate model, or maybe add a confidence head.show more

Midjourney sref + Sora 2 Pro is the sauce.... With one Midjourney style image, you can give a specific style for your entire project. I created two different 12-second clips and edited them together. Some details aren’t fully consistent, like the iPod or AirPods because the clips were made separately from a single image (Character in a specific style). It could be fixed in post-production, but that would take more time, and this was more of an experimental test. It would be great to add the actual product image with the current one to maintain product consistency. I feel like if there were a way to add 2–4 images into this workflow, it could open up a lot more possibilities and consistency. With an API, it could be possible. Or let’s see what Veo 3.1 has to offer.show more

Create a short film like this in just 1... minute with GPT Image 2.0 + Seedance 2.0. GPT Image 2.0 can naturally combine multiple photos into one single image, while Seedance 2.0 can use that image as a reference to automatically separate the scenes, generate a coherent video sequence, and add suitable background music. This workflow greatly improves the overall creative efficiency. When using this method, simply provide the merged image as a reference for Seedance 2.0 and briefly describe each scene with a simple prompt. This can significantly increase the success rate of the final video. All of the above was created on GPT Image Prompt: Seedance Prompt:show more

Colmap 4.0 was very recently released, so it inspired... me to do some work to better understand it and its new capabilities with Rerun. I want to really understand how Colmap, and in particular, pycolmap, works outside of just calling it via the CLI. So my goal is to use the low-level pycolmap API to log every part of the pipeline. The explicit goal is to have an alternative to the SQLite database that I can utilize. Instead of SQLite, I want to try logging everything directly to rerun and use RRD. This means I can have deep inspectability and still save the features/matches/2D view geometry, but be able to view it directly in rerun. I think this is one of the superpowers that rerun provides; data and visualizations are deeply integrated. As I'm often working with sequential data (videos), I'm going to specifically focus on four things: 1. Monocular Video Simple: Calls high-level APIs such as pycolmap.extract_features, pycolmap.match_sequential, pycolmap.incremental_mapping. These are basically identical to the CLI options and provide a good baseline. 2. Monocular Video Streamed: Take the above high-level APIs and break them down to their iterator version, logging each component in a streamed manner. This way, I can stream the intermediate features to rerun while the extraction/matching/mapping is happening. 3. Rig with unknown calibration: <- WHAT THE VIDEO SHOWS This is probably the most interesting version and the first one I've been working on. It allows one to set a rig between known sensors, such as in VR/AR devices, leading to much better reconstructions with multiple cameras. This is the case where we don't know the calibration a priori, so we have to run a reconstruction twice: once as a normal Colmap reconstruction with no rig constraints, use this to generate the constraints, and then do it again with the newly found rig. 4. Rig with known calibration: This is the RoboCap example, where we have a pre-calibrated set of sensors, so we don't need to run the two reconstructions and also gain better matching between cameras, both spatially and temporally. Again, this leads to a much better reconstruction! Along with all this, GLOMAP has become a first-class global mapper, making it super easy to use directly within pycolmap! I'm excited to do more with this and compare it to things like pycuvslam, vipe, and other alternatives.show more

From product image to video with just one tool... - Dzine As you may have noticed, this is one of my favorite tools. It is also very underrated, as probably 50% of my tutorials include some workflow. I was testing the new image-to-video option today, and I love it. Step - by step guide in comments 🔽 I can do 95% of a workflow without switching between apps. Image generation, Image to image with style reference, background removal, background generation, and 2 frames image to video. The only other app I have been using for this video is CapCut so that I can stitch it together. Step by step 🔽show more

Everyone's sleeping on image-to-3D AI models. They can make... your app look incredibly unique, with just a little effort. Here's how. This is my calorie tracker, built in a week with nothing but prompting. Just Claude Code + a couple APIs. The visuals are all AI-generated. I'll be sharing the full workflow + all the crazy technical stuff Claude and I did to make this work, so nobody has to struggle through it like me. Deep dive coming soon! Till then, this is the high-level idea: 1. Get a clean image of the food (or whatever your asset is) - In my app, the user describes foods via text, or attaches images (or both) - If text, an LLM extracts the food description and formats it into a specific prompt I tuned for this design, and we generate an image using Z-Image Turbo through fal - If image, we do the same thing but with FLUX.2 [dev] to edit the user image into our reference design - Originally, both used Google Nano Banana, but switching to open models cut costs and latency a ton 2. Gaussian splatting (2D image → 3D model) - I tried various 2D-to-3D options on fal and ended up with TripoSplat as my preferred balance of speed, cost, latency; this turns an image into a 3D model that looks super high quality (link below) - The app displays the 2D image while our backend generates the 3D splat - We "groom" the splat to reduce size and load time by culling low-opacity/scale points 3. Render efficiently on device Originally, it looked great but ran at 10 FPS. Getting to 120 FPS was a crazy journey. TL;DR: - SwiftUI had to go; it forced us to render each asset in independent MTKViews, which wasn't workable - Instead, we composite every dish into one full-bleed CAMetalLayer using MetalSplatter (link below) - We had to make some optimizations within MetalSplatter's code too, to reduce the overhead of sorting points per render Then I added some finishing touches like the subtle rotation and parallax as they move around. I think it turned out pretty cool :) Overall, this took some effort, but we still got it done in less than a day. Hopefully your agent can follow in the footsteps of mine and do it much faster. Keep an eye out for the bigger writeup, which'll give your agent everything it needs. If you have any questions, drop em below!show more

It is believed that chanting this stotram three times... a day, for six months, fulfils the wishes of devotees. The Yantrodharaka Hanumat Stotram is a hymn in the praise of Lord Hanuman written by Shri Vyasaraja (Vyasatirtha). One day, when sage Vyasaraja was meditating on the banks of River Tungabhadra, he obtained a vision of Lord Hanuman. In order to capture this vision, he etched an image of a monkey on a rock. To his great surprise, the monkey from the illustration came alive. So, he etched an image once again. But, a live monkey materialised from the image once more. So, he repeated this process twelve times. After the twelfth time, he drew a monkey and encased it within a geometric pattern (a yantra) so that it wouldn't incarnate. Hence, the idol came to be known as the 'Yantrodharaka Hanuman' or the 'Hanuman held in the yantra'. Today, one finds a temple around this idol, known as the Yantrodharaka Hanuman Temple, near Hampi.show more

MiniMax H3 Instead of sharing the prompts for each... of these videos, I thought it would be more useful to share how I created that prompts. All of the videos were generated with text-to-video. First, find an image with the kind of scene, composition and mood you want to recreate. I used a few YouTube playlist thumbnails as references but Pinterest is also a great place to find inspiration. You can even use your own old or nostalgic photographs. Then upload the image to ChatGPT and ask it to describe the scene. The description it gives you can essentially become your text-to-video prompt. From there, you can generate completely new scenes with a similar composition, atmosphere and cinematic language. You can of course use the reference image directly with image-to-video or as a first frame. But if the original image isn't yours, I prefer using it only as visual inspiration and recreating the scene through text-to-video. This is the prompt I use with ChatGPT: "Describe the scene in this image in English, focusing primarily on what is happening, the characters, their actions and body language, the setting and the overall atmosphere. Also briefly describe the composition, framing, camera angle, approximate lens choice, lighting, color palette and cinematic aesthetic. Keep it concise and scene-focused rather than overly technical."show more

Gm here’s the next piece I’ll be minting on... SR! I created this video with 245 images I shot in northern Norway over a year ago. I wasn’t that motivated to remove satellites & shooting stars from every image, so they just sat on my hard drive until yesterday. Took me an entire day to edit and create this video, but I’m really glad I finally took the time to do it! Hope someone vibes with it ✨show more

The whole process of putting together the "I hate... cowboys" record and the "go down singin'" record was a long two years of figuring out how do something I've never done before. All the songs were written on guitar and a depth I haven't really hit before. This is the final piece of that puzzle. "The fireside sessions" are both those albums together, all acoustic, featuring the songwriters from both those records. This time in my career ends with both album covers together with me and my dad standing next to each other. It's the first time I've ever been able to see a picture of us as two grown men. I'm moving on to another album early next year, but I hope y'all love these versions of the songs, as they were written. The bluebird last night was a special way to celebrate the ending of these last two years, and the start of something new👏🏼show more

OK, I have a definitive word on the CJ... Abrams play from today's Pittsburgh Pirates at Washington Nationals game after talking with Elias Sports Bureau on this. This play will stay as a sacrifice fly. The originial ruling of NOT a sacrifice fly was for the exact same reason that I thought, which is that the infielder is not running into the outfield, which is year's past would have been correct As it was explained to me, in past years, an infielder had to be running almost in a straight line towards the outfield wall to be considered "running in the outfield". Here, since he is running, and he ends up further away from home (157 feet) than when he started (145 feet), this is going to count as a sacrifice fly. That definition is changing, in part from this play to help bring greater consistency, and to take some of the guesswork out of it (the argument that he is running into the outfield as opposed to more parallel). Now, folks all the time ask "why doesn't MLB publish the OS Manual" and I always say because it is a living document that can have the wording change, and the wording for this play will be modified to something like "more towards the outfield wall than towards home plate" to eliminate any confusion. The big key to this play is that he was running on a full sprint. Also, and this is helpful for me, but for all fly ball outs that score a run, Elias Saba reviews to ensure consistency. So, yes, it's a sacrifice fly, and now that I have that info from Elias themselves, that sort of settles this one. Sounds like the guidelines for this definition will be changing, either this season, or certainly for next season.show more

let’s create the most dank image library for tap.fun.... i quickly vibed together a frontend so everyone can submit images. i’ll drop some cash for the help: - $50 for the most unique meme - $50 for the craziest pic - top 20 will get added to the taplab contributor tg here’s what i’m looking for: images that can transform any image into a unique new one. think memes, crazy visuals, unique outfits, weird energy, funny shit. no text. single image only (not a grid). multiple characters in one pic is totally fine. how to submit: 1. go to 2. submit your image or meme with a name 3. download the image and post it as a comment here so we can see itshow more

Dagknight technical progress As would be mentioned in a... still unshared post by Michael Sutton, the dagknight effort is split into v0 devnet, v1 testnet and v2 mainnet candidate. I’ve been testing the current v0-based implementation in a small devnet with the help of some testers who run nodes and miners with me. The DK work can be thought of as split into two parts: (1) implementing the actual protocol and (2) wiring it up and using it. The testing and development over the last month has been focused on (2). Obviously, DK is a consensus change for selecting parents. What’s not so obvious is that such a change affects DAA, coinbase, IBD, pruning and a lot more. Each of these areas is very sensitive and requires proper understanding to wire correctly. An important consideration and difference from GD is that DK does not focus on maximizing a property like blue work. So to maintain topological properties of blue work, an independent (free) GD implementation is kept running specifically for maintaining blue work. This allows us to keep using the property for topology. Coloring and blue score use the megachain induced by DK. The wiring around DK as of this posting is in a working state, but still needs to be reviewed. Next efforts will be focused on protocol specific components, particularly Tie-Breaking and incremental UMC. Attached are some captures from the internal devnet. The dense DAG image is what happens when things related to DAA or other similar consensus parameter causes a node to insist on their POV. The video is a recent snippet of the KGI running on the devnet showing (perhaps not obviously) DK at work. The current “dagknight” branch is now posted on the main repo. A topic in the Public R&D has been opened for Dagknight development.show more

An interesting issue with Tesla Robotaxi where it took... us to a Starbucks, but the Google data had the incorrect location. At drop off we were 0.2 miles from the Starbucks, so we had a short walk. Is there a way the Tesla AI team could add functionality in the app so riders can update map info to correct errors or inaccuracies on the underlying map data and then this propagates to the fleet? Being able to do this with a pin drop on the map instead of having to use an address might make this very easy and user friendly! Or, as a bigger ask, would it be possible for the car to be able to use visual images on its own to look for a Starbucks sign and on the fly, get us closer and update the map data on its own?show more

excited to announce that I'm officially partnering with the... folks over at Essential Mod ! The Essential team is starting up a new program aimed at supporting mod creators specifically. I've been working with them behind the scenes for a while now to make this program a reality, so I'm very happy to see it come to fruition. I'd really like to see modding become a viable career option for more people, and I think creator programs like this are a step in the right direction. also we'll be collaborating on some cool content in the future so stay tuned!show more