Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Depth-aware light injection in TypeGPU I got a 448x448 monocular depth model down to ~8 ms on my M4 Pro across ~250 dispatches, which is fast enough to use in realtime :D Since the inference is written directly in TypeGPU, I can just feed the depth buffer straight into... show more

Konrad Reczko

7,636 subscribers

10,104,013 views • 1 month ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

50 Comments

Konrad Reczko1 month ago

Here are some fun intermediate textures, the depth and constructed normals

Fernando1 month ago

Really nice. But you shouldn't cast a shadow when the light source is between you and the wall

Konrad Reczko1 month ago

True, but it looks cool :D Doubles as fake godrays

Marc Krenn1 month ago

dope! what depth model are you using?

Konrad Reczko1 month ago

Repacked DepthArt

Marc Krenn1 month ago

thanks, man! :)

dax1 month ago

cool

Rehan Ali Shah1 month ago

Can you share the link of the code you are using for this video. I want to try it.

Konrad Reczko1 month ago

Sure! I’m building it as a TypeGPU example so it will be fully open and runnable just like my selfie segmentation inference

vogel1 month ago

woah this is insane wtf holyshit

グスタボ 🇧🇷1 month ago

can this add dynamic backgrounds in webcam?

Konrad Reczko1 month ago

Yeah, you get raw depth and pixel color you can do whatever you want with it

Mustafa.1 month ago

this is sickk

Rubén1 month ago

You invented Navi! :)

Marc1 month ago

Does this use the TrueDepth camera? VisionCamera?

Konrad Reczko1 month ago

This on web but I will port it to Vision Camera on RN too

Marc1 month ago

Nice! let me know how it goes, VisionCamera gives you depth frames via the CameraDepthOutput:

Konrad Reczko1 month ago

I actually wanted to run my custom network and just use the external texture interface via react-native-webgpu. Worked nicely for me last time :D

Marc1 month ago

wdym exactly? no depth streaming? and which external texture interface do you mean, NativeBuffer?

Konrad Reczko1 month ago

Yup, exactly, I just take the raw camera info and run the inference via TypeGPU compute shaders

Marc1 month ago

ah yes that is the perfect approach for stuff like this. if you build this with VisionCamera can you show me the code? :)

Konrad Reczko1 month ago

Of course, I love the lib btw, I can run code on the camera frames in a way that never reaches JS outside of native handles, brilliant

Sam Sheffer1 month ago

whoa

Indra551 month ago

Dayummm!! Where's the code

Konrad Reczko1 month ago

Soon, I have to clean it up and handle licensing the weights

Indra551 month ago

Waiting!!

ahmet1 month ago

wtf have you built?!?!?!

Gyan Jara Hatke1 month ago

😳😍👌

Kacper Kapuściak1 month ago

what kind of sorcery is this

Ackerman1 month ago

yoo this is so cool, is it opensource?

Konrad Reczko1 month ago

Soon (it will be joining the family of TypeGPU examples I will post an update when it’s ready

John Kim1 month ago

very cool. I can see it being very useful for live streaming.

Google @ Home1 month ago

@grok_owner if you could plz implement this in your livestream to have a fairy fly around you the chat would appreciate it

Michał Wierzbiński1 month ago

Brought some stable fluids to the party. Can confirm: TypeGPU is, ineed, fire 🔥🔥

gfodor.id1 month ago

Wow, definitely gonna check this model out

Atom the UiUx Designer1 month ago

Sorcery

Artur Chakhvadze1 month ago

This is so sick!!!

Joseph Azar1 month ago

that's smooth 🤘

taf 🝮1 month ago

@sivori Amazing!

am.will1 month ago

the lighting physics are the most impressive part. very cool.

Dave Katague1 month ago

dude you are fucking brilliant

adyingdeath1 month ago

This project is super cool. Like a AR game haha.

Aman Gupta1 month ago

Can we use lidar sensor on iphone for accurate depth ??

Konrad Reczko1 month ago

That would be amazing, on web I doubt it but on RN maybe

Duncan1 month ago

wizardry

PDF1 month ago

The shadow effect annoys me, when the hand is between the camera and the light source the light would be directly hitting the wall there should not be a shadow casted from the hand

Konrad Reczko1 month ago

Yeah I get that a lot, good news is the shadow is togglable and I already took the feedback and tweaked the shadows so they are less offensive while still getting the “godray” effect which I like

PDF1 month ago

Niceee, all the best!

Fred Wanders1 month ago

I ve literally been working on something like this for web design. Your implementation is brilliant

Shweta Bharadwaj1 month ago

brilliant stuff, konrad!

Related Videos

You can do some pretty fun realtime VFX once the depth output is just another (Type)GPU buffer :D I basically yoinked my old 3D fire sim (that is actually a fluid sim but do not tell anyone) and plugged it into the depth pipeline together with hand tracking Everything here is built from scratch in TypeGPU, so inference, simulation and rendering can share GPU resources and interleave passes directly. That’s 2 NN inferences, a full 3D fluid sim, relighting and rendering in one command encoder, still under 16 ms with no CPU sync in between

You can do some pretty fun realtime VFX once the depth output is just another (Type)GPU buffer :D I basically yoinked my old 3D fire sim (that is actually a fluid sim but do not tell anyone) and plugged it into the depth pipeline together with hand tracking Everything here is built from scratch in TypeGPU, so inference, simulation and rendering can share GPU resources and interleave passes directly. That’s 2 NN inferences, a full 3D fluid sim, relighting and rendering in one command encoder, still under 16 ms with no CPU sync in between

Konrad Reczko

27,182 views • 1 month ago

TypeGPU + ruNNtime + Jev TypeSafe AI is a very fun combo :D ruNNtime gives me efficient local inference, TypeGPU lets inference and rendering share GPU resources directly with zero copy. That’s 3 separate NN inferences plus rendering, all happening in realtime Since we control the pipeline, Jev can just sit in the middle and add the semantic bit. camera + mic → Moonshine + YOLO26 + DepthART → Jev → lights, shadows and bloom

TypeGPU + ruNNtime + Jev TypeSafe AI is a very fun combo :D ruNNtime gives me efficient local inference, TypeGPU lets inference and rendering share GPU resources directly with zero copy. That’s 3 separate NN inferences plus rendering, all happening in realtime Since we control the pipeline, Jev can just sit in the middle and add the semantic bit. camera + mic → Moonshine + YOLO26 + DepthART → Jev → lights, shadows and bloom

Konrad Reczko

90,251 views • 18 days ago

Anybody remember this game? Just got the game and going to break it down & go in-depth like you’ve never seen before on my YouTube Page The game that got us to #1 in the country! 2007 MU vs ku game at Arrowhead.

Anybody remember this game? Just got the game and going to break it down & go in-depth like you’ve never seen before on my YouTube Page The game that got us to #1 in the country! 2007 MU vs ku game at Arrowhead.

Chase Daniel

228,539 views • 2 years ago

Thank you all for the kind words under the demo I posted :D The demo is now officially public and open source for you to play with, modify and build on top of. Feel free to tag me if you make something cool with it :) Also thank you Iwo Plaza | TypeGPU for participating in my silly video haha You can run it like any other TypeGPU example here: The source code lives in the TypeGPU monorepo: The repacked weights used by the example live here:

Thank you all for the kind words under the demo I posted :D The demo is now officially public and open source for you to play with, modify and build on top of. Feel free to tag me if you make something cool with it :) Also thank you Iwo Plaza | TypeGPU for participating in my silly video haha You can run it like any other TypeGPU example here: The source code lives in the TypeGPU monorepo: The repacked weights used by the example live here:

Konrad Reczko

336,571 views • 1 month ago

My biggest annoyances with VS Code (and in turn Cursor) is I've never been able to get Find + Multi-Cursor with arrow keys working like I did in Sublime Text I use it a lot to edit lists of text fast I go Cmd+Find "SGD" for example, then Find All, then I literally have 10 cursors on every SGD, I can arrow key through the letters around it, or use Option/Alt-Left or Right to skip through words, or Command-Left or Right to to the beginning or the end of the line Then I can use Shift to select I'm super fast at editing like that but none of the shortcuts work in VS Code or Cursor similarly The internet says use Cmd+Shift+L to do it, but all that happens in Cursor is I end up in the AI chat and I don't wanna go there

My biggest annoyances with VS Code (and in turn Cursor) is I've never been able to get Find + Multi-Cursor with arrow keys working like I did in Sublime Text I use it a lot to edit lists of text fast I go Cmd+Find "SGD" for example, then Find All, then I literally have 10 cursors on every SGD, I can arrow key through the letters around it, or use Option/Alt-Left or Right to skip through words, or Command-Left or Right to to the beginning or the end of the line Then I can use Shift to select I'm super fast at editing like that but none of the shortcuts work in VS Code or Cursor similarly The internet says use Cmd+Shift+L to do it, but all that happens in Cursor is I end up in the AI chat and I don't wanna go there

@levelsio

177,684 views • 2 years ago

"This is absolutely stacked... I think this is a good enough squad to go deep in lots of competitions." Phil Kitromilides breaks down the incredible squad depth at Barcelona 🔁

"This is absolutely stacked... I think this is a good enough squad to go deep in lots of competitions." Phil Kitromilides breaks down the incredible squad depth at Barcelona 🔁

CBS Sports Golazo ⚽️

17,577 views • 18 days ago

After playing empulse i think its a little disappointing how accessible the movement is. It lacks depth and skill. Any 1 can be moving fast without much effort. Games like titanfall or quake have incredible depth in its movement to the point where you can spend hours (1/3)

After playing empulse i think its a little disappointing how accessible the movement is. It lacks depth and skill. Any 1 can be moving fast without much effort. Games like titanfall or quake have incredible depth in its movement to the point where you can spend hours (1/3)

GetQuakedOn

60,827 views • 3 months ago

I just got this underwater scooter. It is designed for scuba diving, rated to 130 ft depth and 1 hour of continuous use. Also fun to use in the pool. Next step is to get my son scuba certified. Then after that spear fishing.

I just got this underwater scooter. It is designed for scuba diving, rated to 130 ft depth and 1 hour of continuous use. Also fun to use in the pool. Next step is to get my son scuba certified. Then after that spear fishing.

Wall Street Mav

174,759 views • 1 month ago

This is insane!🔥 This is a 720p (2D) video I shot of my kids 15 years ago! I used AI to upscale it, depth-map it, & used the depth map to convert it to 3D SBS, & then converted that to Spatial Video for Apple Vision Pro (MV-HEVC). My son is currently a senior in high school, and my daughter is a sophomore in college. This video has complete spatial depth when viewed in the Apple Vision Pro. Obviously, you can't appreciate that via this post. But it's like I'm there & I can see into this room. Talk about giving you the feels.🥹 This can be done with ANY video. 🤯 I'm working on refining the workflow, as it's a bit cumbersome. I plan to share it, but holy cow! I had to post this.

This is insane!🔥 This is a 720p (2D) video I shot of my kids 15 years ago! I used AI to upscale it, depth-map it, & used the depth map to convert it to 3D SBS, & then converted that to Spatial Video for Apple Vision Pro (MV-HEVC). My son is currently a senior in high school, and my daughter is a sophomore in college. This video has complete spatial depth when viewed in the Apple Vision Pro. Obviously, you can't appreciate that via this post. But it's like I'm there & I can see into this room. Talk about giving you the feels.🥹 This can be done with ANY video. 🤯 I'm working on refining the workflow, as it's a bit cumbersome. I plan to share it, but holy cow! I had to post this.

Blaine Brown

1,468,930 views • 2 years ago

Arteta speaking after the draw to Wolves: “I think we need to go through the pain and you need to go through that looking in the mirror and understanding what the game requires now and the next action is on Sunday. We need to keep all that in our tummies to show it on Sunday.”

Arteta speaking after the draw to Wolves: “I think we need to go through the pain and you need to go through that looking in the mirror and understanding what the game requires now and the next action is on Sunday. We need to keep all that in our tummies to show it on Sunday.”

Connor Humm

60,050 views • 7 months ago

Was farming first encounter of ghosts to get the helmet I was missing (yeah I know it sucked) and I noticed that there are actually A LOT of details and stuff I just never bothered to notice in these play spaces. For example. Did you you know you can actually SEE Ahsa in this area before dropping down into the methane ocean for the first time? It honestly wants me to go back through all the dungeons in the game and take proper time to look through all the stuff in the world. Is that something you guys would be interested in?

Was farming first encounter of ghosts to get the helmet I was missing (yeah I know it sucked) and I noticed that there are actually A LOT of details and stuff I just never bothered to notice in these play spaces. For example. Did you you know you can actually SEE Ahsa in this area before dropping down into the methane ocean for the first time? It honestly wants me to go back through all the dungeons in the game and take proper time to look through all the stuff in the world. Is that something you guys would be interested in?

NAPainter_ GCX 2026 ✈️

154,528 views • 4 months ago

$I was wrong. I used to tell people that I would never go to a medical doctor unless it was for a broken bone, fracture or some form of structural damage. After having an experience like that I decided I really don't wanna go to a hospital or medical doctor because all they're gonna do is put my bone back in place and then put it in a cast for the next couple months and I'm gonna wonder later on why it never feels the same after locking my bone into the same position. It doesn't make sense to me. Animals in the wild break bones all of the time and fully recover, as long as they don't get eaten by a predator. If I ever have any structural damage again in the future or broken bones I'm gonna use the animal method: just fast, lay down in the sun and let nature heal me.$

I was wrong. I used to tell people that I would never go to a medical doctor unless it was for a broken bone, fracture or some form of structural damage. After having an experience like that I decided I really don't wanna go to a hospital or medical doctor because all they're gonna do is put my bone back in place and then put it in a cast for the next couple months and I'm gonna wonder later on why it never feels the same after locking my bone into the same position. It doesn't make sense to me. Animals in the wild break bones all of the time and fully recover, as long as they don't get eaten by a predator. If I ever have any structural damage again in the future or broken bones I'm gonna use the animal method: just fast, lay down in the sun and let nature heal me.

Riley Check

23,107 views • 5 months ago

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,756 views • 2 years ago

.Compute Network just connected with Pod the Squire and turns staked $ZERO into useable OR sellable inference 49.39M $ZERO is currently staked (ATH) The dual mechanic: - Stake $ZERO, earn daily inference rights - Sell unused inference through - Every staked token has a yield path $15K in $ZERO has also been bought back or burned + $15K has been paid out to stakers

.Compute Network just connected with Pod the Squire and turns staked $ZERO into useable OR sellable inference 49.39M $ZERO is currently staked (ATH) The dual mechanic: - Stake $ZERO, earn daily inference rights - Sell unused inference through - Every staked token has a yield path $15K in $ZERO has also been bought back or burned + $15K has been paid out to stakers

MCG

18,129 views • 3 months ago

I’m bullish on $META over the next 6 to 12 months. Long term target: 820. Macro structure is still bullish and we’re sitting in a discount range as we head into earnings. In this video, I’ll break down the setup in depth and walk through why I finally took a position.

I’m bullish on $META over the next 6 to 12 months. Long term target: 820. Macro structure is still bullish and we’re sitting in a discount range as we head into earnings. In this video, I’ll break down the setup in depth and walk through why I finally took a position.

Peter DiCarlo

144,110 views • 2 months ago

🚨Oreshnik Nightmare is COMING 🚨Russia will take action, sooner rather than later. 🚨We are in a very dangerous situation. 🚨🚨🚨🚨🚨🚨🚨🚨🚨 Scott Ritter: If nuclear-armed states provide strike capacity to a non-nuclear state and use that to strike the strategic depth of Russia, Russia will have to use nuclear weapons preemptively against that threat. The Russian government put the Ukrainians on notice that if drones struck Moscow, Kiev would disappear. NATO missiles on Ukrainian soil striking into the Russian depth. The Flamingo, the FP5, is a British-made missile using German guidance equipment, brought to their target by American and NATO intelligence. Without this, Ukraine would not have this weapon. Ukraine's other weapons are using corridors that go through Poland into the Baltics so they can strike, ... in the vicinity of Saint Petersburg or through Kazakhstan so they can strike into the strategic depth.

🚨Oreshnik Nightmare is COMING 🚨Russia will take action, sooner rather than later. 🚨We are in a very dangerous situation. 🚨🚨🚨🚨🚨🚨🚨🚨🚨 Scott Ritter: If nuclear-armed states provide strike capacity to a non-nuclear state and use that to strike the strategic depth of Russia, Russia will have to use nuclear weapons preemptively against that threat. The Russian government put the Ukrainians on notice that if drones struck Moscow, Kiev would disappear. NATO missiles on Ukrainian soil striking into the Russian depth. The Flamingo, the FP5, is a British-made missile using German guidance equipment, brought to their target by American and NATO intelligence. Without this, Ukraine would not have this weapon. Ukraine's other weapons are using corridors that go through Poland into the Baltics so they can strike, ... in the vicinity of Saint Petersburg or through Kazakhstan so they can strike into the strategic depth.

Ignorance, the root and stem of all evil

37,607 views • 4 months ago

Want to use Depth Anything, but need metric depth rather than relative depth? Thrilled to introduce Prompt Depth Anything, a new paradigm for accurate metric depth estimation with up to 4K resolution. 👉Key Message: Depth foundation models like DA have already internalized rich geometric knowledge of the 3D world but lack a proper way to elicit it. Inspired by the success of prompting in LLMs, we propose prompting Depth Anything with metric cues to produce metric depth. This method proves to be very effective when using a low-cost lidar (e.g., iPhone's LiDAR), which is widely available, as prompts. We believe the prompt can generalize to other forms as long as scale information is provided. Prompt Depth Anything offers 1⃣A series of models for iPhone lidars. 2⃣4D reconstruction from monocular videos (captured with iPhone). 3⃣Improved generalization ability for robot manipulation, e.g. Training on cans but generalizing on glasses. 4⃣More detailed depth annotations for the ScanNet++ dataset. The first author is our excellent intern Haotong Lin. Paper: Huggingface: Project Page: Code:

Want to use Depth Anything, but need metric depth rather than relative depth? Thrilled to introduce Prompt Depth Anything, a new paradigm for accurate metric depth estimation with up to 4K resolution. 👉Key Message: Depth foundation models like DA have already internalized rich geometric knowledge of the 3D world but lack a proper way to elicit it. Inspired by the success of prompting in LLMs, we propose prompting Depth Anything with metric cues to produce metric depth. This method proves to be very effective when using a low-cost lidar (e.g., iPhone's LiDAR), which is widely available, as prompts. We believe the prompt can generalize to other forms as long as scale information is provided. Prompt Depth Anything offers 1⃣A series of models for iPhone lidars. 2⃣4D reconstruction from monocular videos (captured with iPhone). 3⃣Improved generalization ability for robot manipulation, e.g. Training on cans but generalizing on glasses. 4⃣More detailed depth annotations for the ScanNet++ dataset. The first author is our excellent intern Haotong Lin. Paper: Huggingface: Project Page: Code:

Bingyi Kang

67,750 views • 1 year ago

Jane Fonda: “This is the first time in the history of the US that an authoritarian regime has broken through. Worldwide historians who study these things all agree it has never happened so fast in any industrialized democracy. He is moving to consolidate very very fast which means right now we have to use the tools we have to stop it”

Jane Fonda: “This is the first time in the history of the US that an authoritarian regime has broken through. Worldwide historians who study these things all agree it has never happened so fast in any industrialized democracy. He is moving to consolidate very very fast which means right now we have to use the tools we have to stop it”

Marco Foster

831,592 views • 6 months ago