Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

MVP of Multiview Video → Camera parameters + 3D keypoints. Visualized with Rerun The basic pipeline as of right now looks like this: 1. Capture 🔴 – Using 4 iPhones and an Insta360 Go. iPhone videos are captured via Final Cut Pro Multicam for easy sync and the exocentric...

pablovelagomez1's profile picture

Pablo Vela

2,458 subscribers

42,785 görüntüleme • 1 yıl önce •via X (Twitter)

SanatBilim & TeknolojiEğitim
LIVE

Anya Rossi• Live Now

Private livecam show

11 Yorum

Pablo Vela profil fotoğrafı
Pablo Vela1 yıl önce

Links (still a work in progress, but wanted to share for folks who want to dig in): 1. Saved RRD visualization – < 2. Multicam ego/exo sync app – < 3. 3D person detection + triangulation – <

Pablo Vela profil fotoğrafı
Pablo Vela1 yıl önce

Posted the wrong link for the RRD! Updated here <

UserInterface profil fotoğrafı
UserInterface4 yıl önce

Need Professional Video Production, Music Videos, Commercials, Graphic Design, or Photo Retouching? We will take your project from concept to completion. #services #creative #DMV

Jianyuan Wang profil fotoğrafı
Jianyuan Wang1 yıl önce

@rerundotio Yeah I guess VGGT cannot predict high-quality depth maps for dynamic multi-view inputs by now.

Pablo Vela profil fotoğrafı
Pablo Vela1 yıl önce

@rerundotio Yeah, still, it performs better than I expected. Thankfully it produces extremely useful camera parameters. I'm sure it wouldn't be too hard to extend to 4D using something like as an additional dataset

Daniel profil fotoğrafı
Daniel1 yıl önce

@rerundotio Very cool

Pablo Vela profil fotoğrafı
Pablo Vela1 yıl önce

@rerundotio Thank you! Slowly but surely 🫡

Neil Nie profil fotoğrafı
Neil Nie1 yıl önce

@rerundotio Impressive pipeline! thanks for sharing!

Pablo Vela profil fotoğrafı
Pablo Vela1 yıl önce

@rerundotio Thanks for the kind words!

StudioGaltMocap profil fotoğrafı
StudioGaltMocap1 yıl önce

@rerundotio Very cool!

zeke profil fotoğrafı
zeke1 yıl önce

@rerundotio Wow. This has so much potential. Can’t wait to see the progress.

Benzer Videolar

More progress! I now have two Dockerized Gradio | Rerun apps. The first one takes as input a "raw" rrd file that consists of the synchronized egocentric and exocentric MP4 files. This runs the pipeline and produces an "annotated" rrd file. This has the camera parameters, 3D joints, and projected 2D joints (with 6DOF mano soon). The second app takes this "annotated" rrd file and allows for manual labeling. This is a crucial step in addressing any major failures in the pipeline. Right now, it is only the ego view that can be modified. But I'll eventually extend to all. This results in a final "gt" rrd file. From here, the plan is to improve quality and start building a data loop. Excited to start really scaling this. I'm basically going all in on keeping my data stored as Rerun rrd files. As always, I want to emphasize how crucial it is to LOOK AT YOUR data! The rrd format makes it incredibly easy to do so. Getting the data out to use is a bit of a hassle right now, but for me, it's well worth the tradeoff.
1:21

More progress! I now have two Dockerized Gradio | Rerun apps. The first one takes as input a "raw" rrd file that consists of the synchronized egocentric and exocentric MP4 files. This runs the pipeline and produces an "annotated" rrd file. This has the camera parameters, 3D joints, and projected 2D joints (with 6DOF mano soon). The second app takes this "annotated" rrd file and allows for manual labeling. This is a crucial step in addressing any major failures in the pipeline. Right now, it is only the ego view that can be modified. But I'll eventually extend to all. This results in a final "gt" rrd file. From here, the plan is to improve quality and start building a data loop. Excited to start really scaling this. I'm basically going all in on keeping my data stored as Rerun rrd files. As always, I want to emphasize how crucial it is to LOOK AT YOUR data! The rrd format makes it incredibly easy to do so. Getting the data out to use is a bit of a hassle right now, but for me, it's well worth the tradeoff.

Pablo Vela

19,527 görüntüleme • 7 ay önce

Code and data are now online for CameraHMR, our state-of-the-art parametric 3D human pose and shape (HPS) estimation method that will appear at hashtag#3DV2025. There are 4 key contributions that make it so accurate and robust: 1. To get accurate 3D shape and pose as well as good alignment to image features, you need to know the focal length of the camera. To solve this, we train HumanFOV to compute the field of view. 2. We introduce CameraHMR, which integrates HumanFOV into HMR2.0 to exploit the estimated focal length. 3. To get accurate pseudo ground truth (pGT) training data, we compute the focal length for images in 4DHumans dataset and modify SMPLify to take this into account. 4. But SMPLify only uses sparse 2D keypoints, which do not capture body shape. So we train a dense surface keypoint detector, DenseKP, on BEDLAM and run it on 4DHumans, resulting in improved body shape. The resulting method is CamSMPLify. We iterate training CameraHMR and running CamSMPLify on the training set initialized with CameraHMR. This results in much improved pGT for 4DHumans and a SOTA single-image HMR method.
4:38

Code and data are now online for CameraHMR, our state-of-the-art parametric 3D human pose and shape (HPS) estimation method that will appear at hashtag#3DV2025. There are 4 key contributions that make it so accurate and robust: 1. To get accurate 3D shape and pose as well as good alignment to image features, you need to know the focal length of the camera. To solve this, we train HumanFOV to compute the field of view. 2. We introduce CameraHMR, which integrates HumanFOV into HMR2.0 to exploit the estimated focal length. 3. To get accurate pseudo ground truth (pGT) training data, we compute the focal length for images in 4DHumans dataset and modify SMPLify to take this into account. 4. But SMPLify only uses sparse 2D keypoints, which do not capture body shape. So we train a dense surface keypoint detector, DenseKP, on BEDLAM and run it on 4DHumans, resulting in improved body shape. The resulting method is CamSMPLify. We iterate training CameraHMR and running CamSMPLify on the training set initialized with CameraHMR. This results in much improved pGT for 4DHumans and a SOTA single-image HMR method.

Michael Black

21,647 görüntüleme • 1 yıl önce

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.
6:20

Drivable 3D Gaussian Avatars paper page: present Drivable 3D Gaussian Avatars (D3GA), the first 3D controllable model for human bodies rendered with Gaussian splats. Current photorealistic drivable avatars require either accurate 3D registrations during training, dense input images during testing, or both. The ones based on neural radiance fields also tend to be prohibitively slow for telepresence applications. This work uses the recently presented 3D Gaussian Splatting (3DGS) technique to render realistic humans at real-time framerates, using dense calibrated multi-view videos as input. To deform those primitives, we depart from the commonly used point deformation method of linear blend skinning (LBS) and use a classic volumetric deformation method: cage deformations. Given their smaller size, we drive these deformations with joint angles and keypoints, which are more suitable for communication applications. Our experiments on nine subjects with varied body shapes, clothes, and motions obtain higher-quality results than state-of-the-art methods when using the same training and test data.

AK

327,040 görüntüleme • 2 yıl önce

✨ Massive Pipeline Refactor → One Framework for Ego + Exo Datasets, Visualized with Rerun 🚀 After a deep refactoring and cleanup, my entire egocentric/exocentric pipeline is now fully modular. One codebase handles different sensor layouts and generates a unified, multimodal timeseries RRD file that you can open instantly in Rerun. --- The first three datasets that are already supported 1. Assembly101 – 4 ego Quest‑style fisheye cams + 8 exo pinhole cams 2. HO‑Cap – 1 ego HoloLens pinhole cam + 8 exo pinhole cams 3. EgoDex – 1 ego Apple Vision Pro pinhole cam Unified geometry: Each frame now logs _both_ camera intrinsics / extrinsics and COCO Whole-Body 133-kp keypoints in the same stream. Everything is canonicalized at import time, so there’s zero OpenCV vs OpenGL guess-work—Rerun reads it all in the correct coordinate system automatically. --- Why this matters - Consistent schema ✚ live visuals – Rerun’s deep link between data & rendering means every experiment comes with a built‑in viewer. No more ad‑hoc OpenCV/matplotlib hacks just to sanity‑check a dataset. - Multi‑terabyte friendly – The next step is bulk‑ingest these giants into Rerun and wrap them in a Gradio UI for point‑and‑click exploration, as I've already done for EgoDex!
0:26

✨ Massive Pipeline Refactor → One Framework for Ego + Exo Datasets, Visualized with Rerun 🚀 After a deep refactoring and cleanup, my entire egocentric/exocentric pipeline is now fully modular. One codebase handles different sensor layouts and generates a unified, multimodal timeseries RRD file that you can open instantly in Rerun. --- The first three datasets that are already supported 1. Assembly101 – 4 ego Quest‑style fisheye cams + 8 exo pinhole cams 2. HO‑Cap – 1 ego HoloLens pinhole cam + 8 exo pinhole cams 3. EgoDex – 1 ego Apple Vision Pro pinhole cam Unified geometry: Each frame now logs _both_ camera intrinsics / extrinsics and COCO Whole-Body 133-kp keypoints in the same stream. Everything is canonicalized at import time, so there’s zero OpenCV vs OpenGL guess-work—Rerun reads it all in the correct coordinate system automatically. --- Why this matters - Consistent schema ✚ live visuals – Rerun’s deep link between data & rendering means every experiment comes with a built‑in viewer. No more ad‑hoc OpenCV/matplotlib hacks just to sanity‑check a dataset. - Multi‑terabyte friendly – The next step is bulk‑ingest these giants into Rerun and wrap them in a Gradio UI for point‑and‑click exploration, as I've already done for EgoDex!

Pablo Vela

20,836 görüntüleme • 11 ay önce

For 3D pose some use different keypoints, others SMPL and other models. It's a mess! With Neural Localizer Fields, we can choose the output at test time! allowing to train using any. Results are real time and SOTA across the board. István Sárándi
0:25

For 3D pose some use different keypoints, others SMPL and other models. It's a mess! With Neural Localizer Fields, we can choose the output at test time! allowing to train using any. Results are real time and SOTA across the board. István Sárándi

Gerard Pons-Moll

43,459 görüntüleme • 1 yıl önce

Using Nano Banana Pro + Sora, I can now create a short video in just about one minute. I’ve been using Kling for a while to make short videos, and while the results can be great, the workflow has always felt a bit heavy for me. Previously, I had to export the video frame by frame as images, then send those keyframes into Kling’s first/last frame mode to generate the in-betweens. It worked, but it took a lot of time and small steps. Recently I realized I don’t necessarily need to do that anymore: now I can use Nano Banana Pro to generate the images, then send them directly to Sora to create the short video—no need to manually extract a specific frame. Another bonus is that Sora also handles basic sound design for you, which saves me from an extra round of editing and makes the whole process feel much lighter. This idea actually came from a post I saw this morning on Twitter by SD | AI Animation Storyteller, which gave me a new way of thinking about my workflow. I’m still experimenting, but so far it’s been a big improvement in efficiency for my use case. I’ve put the detailed steps I’m using in the comments in case anyone wants to try something similar.
0:24

Using Nano Banana Pro + Sora, I can now create a short video in just about one minute. I’ve been using Kling for a while to make short videos, and while the results can be great, the workflow has always felt a bit heavy for me. Previously, I had to export the video frame by frame as images, then send those keyframes into Kling’s first/last frame mode to generate the in-betweens. It worked, but it took a lot of time and small steps. Recently I realized I don’t necessarily need to do that anymore: now I can use Nano Banana Pro to generate the images, then send them directly to Sora to create the short video—no need to manually extract a specific frame. Another bonus is that Sora also handles basic sound design for you, which saves me from an extra round of editing and makes the whole process feel much lighter. This idea actually came from a post I saw this morning on Twitter by SD | AI Animation Storyteller, which gave me a new way of thinking about my workflow. I’m still experimenting, but so far it’s been a big improvement in efficiency for my use case. I’ve put the detailed steps I’m using in the comments in case anyone wants to try something similar.

underwood

46,400 görüntüleme • 6 ay önce

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.
0:20

Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video Contribution quote from the paper: In summary, our main contributions are • a comprehensive pipeline for reconstructing the shape, appearance, and behavior of real-world garments using Gaussian splatting, • an algorithm for registering garment meshes to multi- view videos with an optimization procedure based on Gaussian splatting, and • a Gaussian Garment representation that combines triangle meshes with Gaussian textures to capture photorealistic appearance and can be used as a fully controllable 3D asset.

MrNeRF

27,277 görüntüleme • 1 yıl önce

We have HOT3D! I've started using Claude to port more datasets into Rerun and exoego-forge. I'd been meaning to bring in the HOT3D dataset from Meta for a while, but with Claude, it's way easier. My goal is to take any egocentric, exocentric, or both datasets and ingest them into a standardized schema. Getting everything into Rerun means we can easily query and transform data via the in-memory OSS server. This lets us generate SQL-like queries such as: "Find me all frames that only contain left hands in the leftmost camera view." Most people think of Rerun as a viewer, but this is the actual superpower. So far we have: 1. HOT3D 2. Hocap 3. UmeTrack 4. Assembly101 5. EgoDex Planning to add more, and with every addition, it gets easier as we build up agent skills and better code examples. Hoping to make it almost fully automatic for adding new datasets. The next few I'm looking at are Harmony4D and Aria Pilot Gen2 After we have enough samples, I'll work on bringing in all the different algorithms I've worked on to transform the data 🙂
0:29

We have HOT3D! I've started using Claude to port more datasets into Rerun and exoego-forge. I'd been meaning to bring in the HOT3D dataset from Meta for a while, but with Claude, it's way easier. My goal is to take any egocentric, exocentric, or both datasets and ingest them into a standardized schema. Getting everything into Rerun means we can easily query and transform data via the in-memory OSS server. This lets us generate SQL-like queries such as: "Find me all frames that only contain left hands in the leftmost camera view." Most people think of Rerun as a viewer, but this is the actual superpower. So far we have: 1. HOT3D 2. Hocap 3. UmeTrack 4. Assembly101 5. EgoDex Planning to add more, and with every addition, it gets easier as we build up agent skills and better code examples. Hoping to make it almost fully automatic for adding new datasets. The next few I'm looking at are Harmony4D and Aria Pilot Gen2 After we have enough samples, I'll work on bringing in all the different algorithms I've worked on to transform the data 🙂

Pablo Vela

35,662 görüntüleme • 2 ay önce

Excited to share this demo from Over the Reality! Watch our Unitree Go2 robodog navigating our office while reconstructing the 3D space in real-time using the VGGT-based foundation vision model. This is a prime example of machine perception in action, turning raw RGB camera feeds into rich, detailed 3D maps! The robodog's RGB cam generates a dense, textured 3D reconstruction via VGGT from a few photograms, capturing nuances like object shapes and surfaces with impressive fidelity (main view). Compare that to the standard LiDAR system (top right), it's sparser, more point-cloud focused, lacking the visual richness. Vision models are closing the gap fast! What's powering this? VGGT, a cutting-edge foundation model for 3D perception, trained on datasets orders of magnitude smaller than our massive OVER 3D maps dataset. Imagine the leap when we apply OVER's scale to VGGT-like transformer based architectures, denser reconstructions, better generalization, revolutionary for robotics, machine perception & AR! Stay tuned for more breakthroughs at the intersection of AI, robotics, and DePIN. We're building the future of Physical AI and Spatial Computing at Over the Reality 🌐 What do you think, ready for robodogs in your world? Drop your thoughts! 🤖🌐
0:25

Excited to share this demo from Over the Reality! Watch our Unitree Go2 robodog navigating our office while reconstructing the 3D space in real-time using the VGGT-based foundation vision model. This is a prime example of machine perception in action, turning raw RGB camera feeds into rich, detailed 3D maps! The robodog's RGB cam generates a dense, textured 3D reconstruction via VGGT from a few photograms, capturing nuances like object shapes and surfaces with impressive fidelity (main view). Compare that to the standard LiDAR system (top right), it's sparser, more point-cloud focused, lacking the visual richness. Vision models are closing the gap fast! What's powering this? VGGT, a cutting-edge foundation model for 3D perception, trained on datasets orders of magnitude smaller than our massive OVER 3D maps dataset. Imagine the leap when we apply OVER's scale to VGGT-like transformer based architectures, denser reconstructions, better generalization, revolutionary for robotics, machine perception & AR! Stay tuned for more breakthroughs at the intersection of AI, robotics, and DePIN. We're building the future of Physical AI and Spatial Computing at Over the Reality 🌐 What do you think, ready for robodogs in your world? Drop your thoughts! 🤖🌐

Over the Reality 🌐

359,500 görüntüleme • 9 ay önce

“This guy will do anything to raise awareness for Covid and Long Covid”. Here are some of the ways I’ve tried. Want to support my efforts to raise awareness and help others? Now is the time. Now is the time to change course. I kicked off fundraising around 19 months ago to... 1. Raise Awareness for Covid and Long Covid. 2. Provide education assistance to families. 3. Provide advocacy assistance. 4. Advocate for education, mitigation, and air filtration. 5. Contact as many political and workforce leaders as I can to request immediate action. If you'd like to donate to my efforts that's great, if you can't that is understandable, and if you just don't want to I would even appreciate a share. I am not raising money to stockpile cash and if we raise a large amount it will go toward setting up a non-profit and hiring a team. That is a far off, long shot goal however so until then I will continue to try to create a job for myself with the help of those who value my work and want to help others. I am not backing down from the goal of becoming a full time advocate for this issue, and I hope to look back on this time as the start of something bigger, and the turning point of when we got enough people to notice and ask for change with us. I will assist people as best I can, I will make as much noise as I can, and I will contact as many policy influencers as I can to help turn the tide, and if you'd like to support the kind of work I'd appreciate it. No obligation friends, if you’re tagged just remove it and I appreciate it if you’ve already donated or even shared. YerAweSum!
2:54

“This guy will do anything to raise awareness for Covid and Long Covid”. Here are some of the ways I’ve tried. Want to support my efforts to raise awareness and help others? Now is the time. Now is the time to change course. I kicked off fundraising around 19 months ago to... 1. Raise Awareness for Covid and Long Covid. 2. Provide education assistance to families. 3. Provide advocacy assistance. 4. Advocate for education, mitigation, and air filtration. 5. Contact as many political and workforce leaders as I can to request immediate action. If you'd like to donate to my efforts that's great, if you can't that is understandable, and if you just don't want to I would even appreciate a share. I am not raising money to stockpile cash and if we raise a large amount it will go toward setting up a non-profit and hiring a team. That is a far off, long shot goal however so until then I will continue to try to create a job for myself with the help of those who value my work and want to help others. I am not backing down from the goal of becoming a full time advocate for this issue, and I hope to look back on this time as the start of something bigger, and the turning point of when we got enough people to notice and ask for change with us. I will assist people as best I can, I will make as much noise as I can, and I will contact as many policy influencers as I can to help turn the tide, and if you'd like to support the kind of work I'd appreciate it. No obligation friends, if you’re tagged just remove it and I appreciate it if you’ve already donated or even shared. YerAweSum!

Keith

92,638 görüntüleme • 2 ay önce

I left the UK to ILORIN to surprise my family after being away for about 2yrs. Nobody knew I was coming except for Ustaz Awon Hoodlums as I felt it’s normal to let at least one person know about my journey. Each frame of this post means so much to me and my family and in a long time this is the happiest I’ve been. Alhamdulillah. Frame 1 : My mom Frame 2: My elder bro Frame 3 : My grandma Frame 4 : My sister who had dreamt 2 days earlier that I arrived and told her husband about it, she kept saying “I said it” 😂
0:12

I left the UK to ILORIN to surprise my family after being away for about 2yrs. Nobody knew I was coming except for Ustaz Awon Hoodlums as I felt it’s normal to let at least one person know about my journey. Each frame of this post means so much to me and my family and in a long time this is the happiest I’ve been. Alhamdulillah. Frame 1 : My mom Frame 2: My elder bro Frame 3 : My grandma Frame 4 : My sister who had dreamt 2 days earlier that I arrived and told her husband about it, she kept saying “I said it” 😂

👳🏾‍♂️Mufti Of Ilorin Online 👳🏾‍♂️

596,568 görüntüleme • 1 yıl önce

Boom! Open source LegoGPT is a building AI, and sure it can be used for Legos but it can also be used for Lego-like building of homes. LegoGPT converts meshes to Lego in one step using 1×1, 1×2, 1×4, 1×6, 1×8, 2×2, 2×4, and 2×6 bricks. Then they evaluate the stability of the design. Finally, they render an image and ask GPT-4o to produce captions to go with the image. This sent to robots and they complete the real-time build. This can absolutely scale and as with Legos, you can use smaller pieces for higher resolution. Testing it detail now with robot assembly. Link:
0:45

Boom! Open source LegoGPT is a building AI, and sure it can be used for Legos but it can also be used for Lego-like building of homes. LegoGPT converts meshes to Lego in one step using 1×1, 1×2, 1×4, 1×6, 1×8, 2×2, 2×4, and 2×6 bricks. Then they evaluate the stability of the design. Finally, they render an image and ask GPT-4o to produce captions to go with the image. This sent to robots and they complete the real-time build. This can absolutely scale and as with Legos, you can use smaller pieces for higher resolution. Testing it detail now with robot assembly. Link:

Brian Roemmele

22,363 görüntüleme • 1 yıl önce

I created a 3D Gaussian Splat of my kitchen using 1/3rd the images it used to take me by using NVIDIA AI Developer's 3DGRUT! I can now use 180 degree fisheye images and ray tracing to make detailed splats. The only reason the scene isn't sharper is because my input images weren't super sharp - when I took the images back in October, I was still learning to use the lens. I plan to make a "first reactions/overview video". Tutorial after that. For reference, this took 206 images and the ultrawide on my iPhone took 608 images to capture. #3D #AEC #Computervision
0:40

I created a 3D Gaussian Splat of my kitchen using 1/3rd the images it used to take me by using NVIDIA AI Developer's 3DGRUT! I can now use 180 degree fisheye images and ray tracing to make detailed splats. The only reason the scene isn't sharper is because my input images weren't super sharp - when I took the images back in October, I was still learning to use the lens. I plan to make a "first reactions/overview video". Tutorial after that. For reference, this took 206 images and the ultrawide on my iPhone took 608 images to capture. #3D #AEC #Computervision

Jonathan Stephens

30,646 görüntüleme • 1 yıl önce

VGGT: Visual Geometry Grounded Transformer TL;DR: Is DUSt3R facing a formidable new rival? Contributions: (1) We introduce VGGT, a large feed-forward transformer that can, given one, a few, or even hundreds of images of a scene, predict all its key 3D attributes - including camera intrinsics and extrinsics, point maps, depth maps, and 3D point tracks - in seconds. (2) We demonstrate that VGGT’s predictions are directly usable, being highly competitive and usually better than those of state-of-the-art methods that use slow post-processing optimization techniques. (3) We also show that when further combined with BA post-processing, VGGT achieves state-of-the-art results across the board, even when compared to methods that specialize in a subset of 3D tasks, often improving quality substantially.
0:28

VGGT: Visual Geometry Grounded Transformer TL;DR: Is DUSt3R facing a formidable new rival? Contributions: (1) We introduce VGGT, a large feed-forward transformer that can, given one, a few, or even hundreds of images of a scene, predict all its key 3D attributes - including camera intrinsics and extrinsics, point maps, depth maps, and 3D point tracks - in seconds. (2) We demonstrate that VGGT’s predictions are directly usable, being highly competitive and usually better than those of state-of-the-art methods that use slow post-processing optimization techniques. (3) We also show that when further combined with BA post-processing, VGGT achieves state-of-the-art results across the board, even when compared to methods that specialize in a subset of 3D tasks, often improving quality substantially.

MrNeRF

29,461 görüntüleme • 1 yıl önce

This is probably the most complex workflow I’ve ever built, only with open-source tools. It took my 4 days. It takes four inputs: author, title, and style; and generates a full visual animated story in one click in ComfyUI . I worked on it for four days. There are still some bugs, but here’s the first preview. Here’s a quick breakdown: - The four inputs are sent to LLMs with precise instructions to generate: first, prompts for images and image modifications; second, prompts for animations; third, prompts for generating music. - All voices are generated from the text and timed precisely, as they determine the length of each animation segment. - The first image and video are generated to serve as the title, but also as the guide for all other images created for the video. - Titles and subtitles are also added automatically in Comfy. - I also developed a lot of custom nodes for minor frame calculations, mostly to match audio and video. - The full system is a large loop that, for each line of text, generates an image and then a video from that image. The loop was the hardest part to build in this workflow, so it can process either a 20-second video or a 2-minute video with the same input. - There are multiple combinations of LLMs that try to understand the text in the best way to provide the best prompts for images and video. - The final video is assembled entirely within ComfyUI. - The music is generated based on the LLM output and matches the exact timing of the full animation. - Done! For reference, this workflow uses a lot of models and only works on an RTX 6000 Pro with plenty of RAM. My goal is not to replace humans, as I’ll try to explain later, this workflow is highly controlled and can be adapted or reworked at any point by real artists! My aim was to create a tool that can animate text in one go, allowing the AI some freedom while keeping a strict flow. I don’t know yet how I’ll share this workflow with people, I still need to polish it properly, but maybe through Patreon. Anyway, I hope you enjoy my research, and let’s always keep pushing further! :)
0:55

This is probably the most complex workflow I’ve ever built, only with open-source tools. It took my 4 days. It takes four inputs: author, title, and style; and generates a full visual animated story in one click in ComfyUI . I worked on it for four days. There are still some bugs, but here’s the first preview. Here’s a quick breakdown: - The four inputs are sent to LLMs with precise instructions to generate: first, prompts for images and image modifications; second, prompts for animations; third, prompts for generating music. - All voices are generated from the text and timed precisely, as they determine the length of each animation segment. - The first image and video are generated to serve as the title, but also as the guide for all other images created for the video. - Titles and subtitles are also added automatically in Comfy. - I also developed a lot of custom nodes for minor frame calculations, mostly to match audio and video. - The full system is a large loop that, for each line of text, generates an image and then a video from that image. The loop was the hardest part to build in this workflow, so it can process either a 20-second video or a 2-minute video with the same input. - There are multiple combinations of LLMs that try to understand the text in the best way to provide the best prompts for images and video. - The final video is assembled entirely within ComfyUI. - The music is generated based on the LLM output and matches the exact timing of the full animation. - Done! For reference, this workflow uses a lot of models and only works on an RTX 6000 Pro with plenty of RAM. My goal is not to replace humans, as I’ll try to explain later, this workflow is highly controlled and can be adapted or reworked at any point by real artists! My aim was to create a tool that can animate text in one go, allowing the AI some freedom while keeping a strict flow. I don’t know yet how I’ll share this workflow with people, I still need to polish it properly, but maybe through Patreon. Anyway, I hope you enjoy my research, and let’s always keep pushing further! :)

Lovis Odin

56,518 görüntüleme • 8 ay önce

First frame + Last frame = Fashion ads in minutes. Here’s how I generated this U.S. Polo sequence using Kling 3.0 + Calico AI: → The Input: A start frame and an end frame to lock in the subject and composition. → The Prompt: Don't just ask for a "cool video." Specifically prompted for "slow cinematic push-in," "shallow depth of field," and "micro-parallax" to get that premium brand-film look. → The Output: Motion that looks like a high-budget fashion shoot. Studio shoots are expensive, but AI is leveling the playing field. Comment 'PROMPT' if you want me to send you all of the prompts and the system I used to build this. (must be following so I can dm you!)
0:21

First frame + Last frame = Fashion ads in minutes. Here’s how I generated this U.S. Polo sequence using Kling 3.0 + Calico AI: → The Input: A start frame and an end frame to lock in the subject and composition. → The Prompt: Don't just ask for a "cool video." Specifically prompted for "slow cinematic push-in," "shallow depth of field," and "micro-parallax" to get that premium brand-film look. → The Output: Motion that looks like a high-budget fashion shoot. Studio shoots are expensive, but AI is leveling the playing field. Comment 'PROMPT' if you want me to send you all of the prompts and the system I used to build this. (must be following so I can dm you!)

David Roberts

25,878 görüntüleme • 3 ay önce

Here’s a sneak peek using Rerun and Gradio for data annotation. It uses Video Depth Anything and Segment Anything 2 under the hood to generate segmentation masks and depth maps/point clouds. More to share next week.
0:30

Here’s a sneak peek using Rerun and Gradio for data annotation. It uses Video Depth Anything and Segment Anything 2 under the hood to generate segmentation masks and depth maps/point clouds. More to share next week.

Pablo Vela

36,719 görüntüleme • 1 yıl önce

This is the first cut of my trailer for my idle Basebuilding RPG with a deckbuilding roguelike adventure, Sector Scavengers! I’m a little over 20 days into development and having a blast! Here’s the set of tools I used for this: 1. Makko AI for gameplay and art 2. ElevenLabs for voiceover 3. Grok for the cut scenes 4. BandM8 for the background music 5. CapCut to bring it all together Starting out as a solo dev at age 40 wasn’t on my dance card but here we are! Thanks to everyone who is following along and supporting!
1:54

This is the first cut of my trailer for my idle Basebuilding RPG with a deckbuilding roguelike adventure, Sector Scavengers! I’m a little over 20 days into development and having a blast! Here’s the set of tools I used for this: 1. Makko AI for gameplay and art 2. ElevenLabs for voiceover 3. Grok for the cut scenes 4. BandM8 for the background music 5. CapCut to bring it all together Starting out as a solo dev at age 40 wasn’t on my dance card but here we are! Thanks to everyone who is following along and supporting!

Sector Scavengers | Wishlist on Steam

48,535 görüntüleme • 2 ay önce

we’ll call this one results of the cut cut began back in late November/early December with the goal of revealing the muscle put on in the previous bulk, idea being to bring the best physique to date all roads culminated at the All Elite Wrestling PPV Revolution on March 9th, this video was the last check in I sent to my coach pre match can safely say both the bulk and cut were a success, as I’m leaner than the end of the previous cut by a fair margin and over 10lbs heavier, meaning I added at least 10lbs of lean muscle to my frame so now we get to call this my new “peak physique” myself/my coach have already put the plan in place for what’s next and I am SO EXCITED to get back in the lab and get back to work this pressure is a privilege this is what I live for
0:36

we’ll call this one results of the cut cut began back in late November/early December with the goal of revealing the muscle put on in the previous bulk, idea being to bring the best physique to date all roads culminated at the All Elite Wrestling PPV Revolution on March 9th, this video was the last check in I sent to my coach pre match can safely say both the bulk and cut were a success, as I’m leaner than the end of the previous cut by a fair margin and over 10lbs heavier, meaning I added at least 10lbs of lean muscle to my frame so now we get to call this my new “peak physique” myself/my coach have already put the plan in place for what’s next and I am SO EXCITED to get back in the lab and get back to work this pressure is a privilege this is what I live for

AEW TNT Champion “The Protostar” Kyle Fletcher

183,597 görüntüleme • 1 yıl önce

From here begins the photoshop process. I don't want to display every part of this - but i like using a lot of vectors and halftones in my peices. Since this is a speedrender, i decided to use a basic halftone background, and use a pre-existing fortnite HDRI for the bg.
1:32

From here begins the photoshop process. I don't want to display every part of this - but i like using a lot of vectors and halftones in my peices. Since this is a speedrender, i decided to use a basic halftone background, and use a pre-existing fortnite HDRI for the bg.

RothStuff! 2/3

13,770 görüntüleme • 2 ay önce