正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Depth-aware light injection in TypeGPU I got a 448x448 monocular depth model down to ~8 ms on my M4 Pro across ~250 dispatches, which is fast enough to use in realtime :D Since the inference is written directly in TypeGPU, I can just feed the depth buffer straight into... show more

Konrad Reczko

7,636 subscribers

10,104,013 次观看 • 1 个月前 •via X (Twitter)

科学技术教育

Anya Rossi• Live Now

Private livecam show

50 条评论

Konrad Reczko 的头像

Konrad Reczko1 个月前

Here are some fun intermediate textures, the depth and constructed normals

Fernando 的头像

Fernando1 个月前

Really nice. But you shouldn't cast a shadow when the light source is between you and the wall

Konrad Reczko 的头像

Konrad Reczko1 个月前

True, but it looks cool :D Doubles as fake godrays

Marc Krenn 的头像

Marc Krenn1 个月前

dope! what depth model are you using?

Konrad Reczko 的头像

Konrad Reczko1 个月前

Repacked DepthArt

Marc Krenn 的头像

Marc Krenn1 个月前

thanks, man! :)

dax 的头像

dax1 个月前

cool

Rehan Ali Shah 的头像

Rehan Ali Shah1 个月前

Can you share the link of the code you are using for this video. I want to try it.

Konrad Reczko 的头像

Konrad Reczko1 个月前

Sure! I’m building it as a TypeGPU example so it will be fully open and runnable just like my selfie segmentation inference

vogel 的头像

vogel1 个月前

woah this is insane wtf holyshit

グスタボ 🇧🇷 的头像

グスタボ 🇧🇷1 个月前

can this add dynamic backgrounds in webcam?

Konrad Reczko 的头像

Konrad Reczko1 个月前

Yeah, you get raw depth and pixel color you can do whatever you want with it

Mustafa. 的头像

Mustafa.1 个月前

this is sickk

Rubén 的头像

Rubén1 个月前

You invented Navi! :)

Marc 的头像

Marc1 个月前

Does this use the TrueDepth camera? VisionCamera?

Konrad Reczko 的头像

Konrad Reczko1 个月前

This on web but I will port it to Vision Camera on RN too

Marc 的头像

Marc1 个月前

Nice! let me know how it goes, VisionCamera gives you depth frames via the CameraDepthOutput:

Konrad Reczko 的头像

Konrad Reczko1 个月前

I actually wanted to run my custom network and just use the external texture interface via react-native-webgpu. Worked nicely for me last time :D

Marc 的头像

Marc1 个月前

wdym exactly? no depth streaming? and which external texture interface do you mean, NativeBuffer?

Konrad Reczko 的头像

Konrad Reczko1 个月前

Yup, exactly, I just take the raw camera info and run the inference via TypeGPU compute shaders

Marc 的头像

Marc1 个月前

ah yes that is the perfect approach for stuff like this. if you build this with VisionCamera can you show me the code? :)

Konrad Reczko 的头像

Konrad Reczko1 个月前

Of course, I love the lib btw, I can run code on the camera frames in a way that never reaches JS outside of native handles, brilliant

Sam Sheffer 的头像

Sam Sheffer1 个月前

whoa

Indra55 的头像

Indra551 个月前

Dayummm!! Where's the code

Konrad Reczko 的头像

Konrad Reczko1 个月前

Soon, I have to clean it up and handle licensing the weights

Indra55 的头像

Indra551 个月前

Waiting!!

ahmet 的头像

ahmet1 个月前

wtf have you built?!?!?!

Gyan Jara Hatke 的头像

Gyan Jara Hatke1 个月前

😳😍👌

Kacper Kapuściak 的头像

Kacper Kapuściak1 个月前

what kind of sorcery is this

Ackerman 的头像

Ackerman1 个月前

yoo this is so cool, is it opensource?

Konrad Reczko 的头像

Konrad Reczko1 个月前

Soon (it will be joining the family of TypeGPU examples I will post an update when it’s ready

John Kim 的头像

John Kim1 个月前

very cool. I can see it being very useful for live streaming.

Google @ Home 的头像

Google @ Home1 个月前

@grok_owner if you could plz implement this in your livestream to have a fairy fly around you the chat would appreciate it

Michał Wierzbiński 的头像

Michał Wierzbiński1 个月前

Brought some stable fluids to the party. Can confirm: TypeGPU is, ineed, fire 🔥🔥

gfodor.id 的头像

gfodor.id1 个月前

Wow, definitely gonna check this model out

Atom the UiUx Designer 的头像

Atom the UiUx Designer1 个月前

Sorcery

Artur Chakhvadze 的头像

Artur Chakhvadze1 个月前

This is so sick!!!

Joseph Azar 的头像

Joseph Azar1 个月前

that's smooth 🤘

taf 🝮 的头像

taf 🝮1 个月前

@sivori Amazing!

am.will 的头像

am.will1 个月前

the lighting physics are the most impressive part. very cool.

Dave Katague 的头像

Dave Katague1 个月前

dude you are fucking brilliant

adyingdeath 的头像

adyingdeath1 个月前

This project is super cool. Like a AR game haha.

Aman Gupta 的头像

Aman Gupta1 个月前

Can we use lidar sensor on iphone for accurate depth ??

Konrad Reczko 的头像

Konrad Reczko1 个月前

That would be amazing, on web I doubt it but on RN maybe

Duncan 的头像

Duncan1 个月前

wizardry

PDF 的头像

PDF1 个月前

The shadow effect annoys me, when the hand is between the camera and the light source the light would be directly hitting the wall there should not be a shadow casted from the hand

Konrad Reczko 的头像

Konrad Reczko1 个月前

Yeah I get that a lot, good news is the shadow is togglable and I already took the feedback and tweaked the shadows so they are less offensive while still getting the “godray” effect which I like

PDF 的头像

PDF1 个月前

Niceee, all the best!

Fred Wanders 的头像

Fred Wanders1 个月前

I ve literally been working on something like this for web design. Your implementation is brilliant

Shweta Bharadwaj 的头像

Shweta Bharadwaj1 个月前

brilliant stuff, konrad!

相关视频

You can do some pretty fun realtime VFX once the depth output is just another (Type)GPU buffer :D I basically yoinked my old 3D fire sim (that is actually a fluid sim but do not tell anyone) and plugged it into the depth pipeline together with hand tracking Everything here is built from scratch in TypeGPU, so inference, simulation and rendering can share GPU resources and interleave passes directly. That’s 2 NN inferences, a full 3D fluid sim, relighting and rendering in one command encoder, still under 16 ms with no CPU sync in between

You can do some pretty fun realtime VFX once the depth output is just another (Type)GPU buffer :D I basically yoinked my old 3D fire sim (that is actually a fluid sim but do not tell anyone) and plugged it into the depth pipeline together with hand tracking Everything here is built from scratch in TypeGPU, so inference, simulation and rendering can share GPU resources and interleave passes directly. That’s 2 NN inferences, a full 3D fluid sim, relighting and rendering in one command encoder, still under 16 ms with no CPU sync in between

Konrad Reczko

27,182 次观看 • 1 个月前

TypeGPU + ruNNtime + Jev TypeSafe AI is a very fun combo :D ruNNtime gives me efficient local inference, TypeGPU lets inference and rendering share GPU resources directly with zero copy. That’s 3 separate NN inferences plus rendering, all happening in realtime Since we control the pipeline, Jev can just sit in the middle and add the semantic bit. camera + mic → Moonshine + YOLO26 + DepthART → Jev → lights, shadows and bloom

TypeGPU + ruNNtime + Jev TypeSafe AI is a very fun combo :D ruNNtime gives me efficient local inference, TypeGPU lets inference and rendering share GPU resources directly with zero copy. That’s 3 separate NN inferences plus rendering, all happening in realtime Since we control the pipeline, Jev can just sit in the middle and add the semantic bit. camera + mic → Moonshine + YOLO26 + DepthART → Jev → lights, shadows and bloom

Konrad Reczko

90,251 次观看 • 18 天前

Anybody remember this game? Just got the game and going to break it down & go in-depth like you’ve never seen before on my YouTube Page The game that got us to #1 in the country! 2007 MU vs ku game at Arrowhead.

Anybody remember this game? Just got the game and going to break it down & go in-depth like you’ve never seen before on my YouTube Page The game that got us to #1 in the country! 2007 MU vs ku game at Arrowhead.

Chase Daniel

228,539 次观看 • 2 年前

Thank you all for the kind words under the demo I posted :D The demo is now officially public and open source for you to play with, modify and build on top of. Feel free to tag me if you make something cool with it :) Also thank you Iwo Plaza | TypeGPU for participating in my silly video haha You can run it like any other TypeGPU example here: The source code lives in the TypeGPU monorepo: The repacked weights used by the example live here:

Thank you all for the kind words under the demo I posted :D The demo is now officially public and open source for you to play with, modify and build on top of. Feel free to tag me if you make something cool with it :) Also thank you Iwo Plaza | TypeGPU for participating in my silly video haha You can run it like any other TypeGPU example here: The source code lives in the TypeGPU monorepo: The repacked weights used by the example live here:

Konrad Reczko

336,571 次观看 • 1 个月前

My biggest annoyances with VS Code (and in turn Cursor) is I've never been able to get Find + Multi-Cursor with arrow keys working like I did in Sublime Text I use it a lot to edit lists of text fast I go Cmd+Find "SGD" for example, then Find All, then I literally have 10 cursors on every SGD, I can arrow key through the letters around it, or use Option/Alt-Left or Right to skip through words, or Command-Left or Right to to the beginning or the end of the line Then I can use Shift to select I'm super fast at editing like that but none of the shortcuts work in VS Code or Cursor similarly The internet says use Cmd+Shift+L to do it, but all that happens in Cursor is I end up in the AI chat and I don't wanna go there

My biggest annoyances with VS Code (and in turn Cursor) is I've never been able to get Find + Multi-Cursor with arrow keys working like I did in Sublime Text I use it a lot to edit lists of text fast I go Cmd+Find "SGD" for example, then Find All, then I literally have 10 cursors on every SGD, I can arrow key through the letters around it, or use Option/Alt-Left or Right to skip through words, or Command-Left or Right to to the beginning or the end of the line Then I can use Shift to select I'm super fast at editing like that but none of the shortcuts work in VS Code or Cursor similarly The internet says use Cmd+Shift+L to do it, but all that happens in Cursor is I end up in the AI chat and I don't wanna go there

@levelsio

177,684 次观看 • 2 年前

"This is absolutely stacked... I think this is a good enough squad to go deep in lots of competitions." Phil Kitromilides breaks down the incredible squad depth at Barcelona 🔁

"This is absolutely stacked... I think this is a good enough squad to go deep in lots of competitions." Phil Kitromilides breaks down the incredible squad depth at Barcelona 🔁

CBS Sports Golazo ⚽️

17,577 次观看 • 18 天前

After playing empulse i think its a little disappointing how accessible the movement is. It lacks depth and skill. Any 1 can be moving fast without much effort. Games like titanfall or quake have incredible depth in its movement to the point where you can spend hours (1/3)

After playing empulse i think its a little disappointing how accessible the movement is. It lacks depth and skill. Any 1 can be moving fast without much effort. Games like titanfall or quake have incredible depth in its movement to the point where you can spend hours (1/3)

GetQuakedOn

60,827 次观看 • 3 个月前

I just got this underwater scooter. It is designed for scuba diving, rated to 130 ft depth and 1 hour of continuous use. Also fun to use in the pool. Next step is to get my son scuba certified. Then after that spear fishing.

I just got this underwater scooter. It is designed for scuba diving, rated to 130 ft depth and 1 hour of continuous use. Also fun to use in the pool. Next step is to get my son scuba certified. Then after that spear fishing.

Wall Street Mav

174,759 次观看 • 1 个月前

This is insane!🔥 This is a 720p (2D) video I shot of my kids 15 years ago! I used AI to upscale it, depth-map it, & used the depth map to convert it to 3D SBS, & then converted that to Spatial Video for Apple Vision Pro (MV-HEVC). My son is currently a senior in high school, and my daughter is a sophomore in college. This video has complete spatial depth when viewed in the Apple Vision Pro. Obviously, you can't appreciate that via this post. But it's like I'm there & I can see into this room. Talk about giving you the feels.🥹 This can be done with ANY video. 🤯 I'm working on refining the workflow, as it's a bit cumbersome. I plan to share it, but holy cow! I had to post this.

This is insane!🔥 This is a 720p (2D) video I shot of my kids 15 years ago! I used AI to upscale it, depth-map it, & used the depth map to convert it to 3D SBS, & then converted that to Spatial Video for Apple Vision Pro (MV-HEVC). My son is currently a senior in high school, and my daughter is a sophomore in college. This video has complete spatial depth when viewed in the Apple Vision Pro. Obviously, you can't appreciate that via this post. But it's like I'm there & I can see into this room. Talk about giving you the feels.🥹 This can be done with ANY video. 🤯 I'm working on refining the workflow, as it's a bit cumbersome. I plan to share it, but holy cow! I had to post this.

Blaine Brown

1,468,930 次观看 • 2 年前

Arteta speaking after the draw to Wolves: “I think we need to go through the pain and you need to go through that looking in the mirror and understanding what the game requires now and the next action is on Sunday. We need to keep all that in our tummies to show it on Sunday.”

Arteta speaking after the draw to Wolves: “I think we need to go through the pain and you need to go through that looking in the mirror and understanding what the game requires now and the next action is on Sunday. We need to keep all that in our tummies to show it on Sunday.”

Connor Humm

60,050 次观看 • 7 个月前

Was farming first encounter of ghosts to get the helmet I was missing (yeah I know it sucked) and I noticed that there are actually A LOT of details and stuff I just never bothered to notice in these play spaces. For example. Did you you know you can actually SEE Ahsa in this area before dropping down into the methane ocean for the first time? It honestly wants me to go back through all the dungeons in the game and take proper time to look through all the stuff in the world. Is that something you guys would be interested in?

Was farming first encounter of ghosts to get the helmet I was missing (yeah I know it sucked) and I noticed that there are actually A LOT of details and stuff I just never bothered to notice in these play spaces. For example. Did you you know you can actually SEE Ahsa in this area before dropping down into the methane ocean for the first time? It honestly wants me to go back through all the dungeons in the game and take proper time to look through all the stuff in the world. Is that something you guys would be interested in?

NAPainter_ GCX 2026 ✈️

154,528 次观看 • 4 个月前

$I was wrong. I used to tell people that I would never go to a medical doctor unless it was for a broken bone, fracture or some form of structural damage. After having an experience like that I decided I really don't wanna go to a hospital or medical doctor because all they're gonna do is put my bone back in place and then put it in a cast for the next couple months and I'm gonna wonder later on why it never feels the same after locking my bone into the same position. It doesn't make sense to me. Animals in the wild break bones all of the time and fully recover, as long as they don't get eaten by a predator. If I ever have any structural damage again in the future or broken bones I'm gonna use the animal method: just fast, lay down in the sun and let nature heal me.$

I was wrong. I used to tell people that I would never go to a medical doctor unless it was for a broken bone, fracture or some form of structural damage. After having an experience like that I decided I really don't wanna go to a hospital or medical doctor because all they're gonna do is put my bone back in place and then put it in a cast for the next couple months and I'm gonna wonder later on why it never feels the same after locking my bone into the same position. It doesn't make sense to me. Animals in the wild break bones all of the time and fully recover, as long as they don't get eaten by a predator. If I ever have any structural damage again in the future or broken bones I'm gonna use the animal method: just fast, lay down in the sun and let nature heal me.

Riley Check

23,107 次观看 • 5 个月前

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,756 次观看 • 2 年前

.Compute Network just connected with Pod the Squire and turns staked $ZERO into useable OR sellable inference 49.39M $ZERO is currently staked (ATH) The dual mechanic: - Stake $ZERO, earn daily inference rights - Sell unused inference through - Every staked token has a yield path $15K in $ZERO has also been bought back or burned + $15K has been paid out to stakers

.Compute Network just connected with Pod the Squire and turns staked $ZERO into useable OR sellable inference 49.39M $ZERO is currently staked (ATH) The dual mechanic: - Stake $ZERO, earn daily inference rights - Sell unused inference through - Every staked token has a yield path $15K in $ZERO has also been bought back or burned + $15K has been paid out to stakers

MCG

18,129 次观看 • 3 个月前

I’m bullish on $META over the next 6 to 12 months. Long term target: 820. Macro structure is still bullish and we’re sitting in a discount range as we head into earnings. In this video, I’ll break down the setup in depth and walk through why I finally took a position.

I’m bullish on $META over the next 6 to 12 months. Long term target: 820. Macro structure is still bullish and we’re sitting in a discount range as we head into earnings. In this video, I’ll break down the setup in depth and walk through why I finally took a position.

Peter DiCarlo

144,110 次观看 • 2 个月前

🚨Oreshnik Nightmare is COMING 🚨Russia will take action, sooner rather than later. 🚨We are in a very dangerous situation. 🚨🚨🚨🚨🚨🚨🚨🚨🚨 Scott Ritter: If nuclear-armed states provide strike capacity to a non-nuclear state and use that to strike the strategic depth of Russia, Russia will have to use nuclear weapons preemptively against that threat. The Russian government put the Ukrainians on notice that if drones struck Moscow, Kiev would disappear. NATO missiles on Ukrainian soil striking into the Russian depth. The Flamingo, the FP5, is a British-made missile using German guidance equipment, brought to their target by American and NATO intelligence. Without this, Ukraine would not have this weapon. Ukraine's other weapons are using corridors that go through Poland into the Baltics so they can strike, ... in the vicinity of Saint Petersburg or through Kazakhstan so they can strike into the strategic depth.

🚨Oreshnik Nightmare is COMING 🚨Russia will take action, sooner rather than later. 🚨We are in a very dangerous situation. 🚨🚨🚨🚨🚨🚨🚨🚨🚨 Scott Ritter: If nuclear-armed states provide strike capacity to a non-nuclear state and use that to strike the strategic depth of Russia, Russia will have to use nuclear weapons preemptively against that threat. The Russian government put the Ukrainians on notice that if drones struck Moscow, Kiev would disappear. NATO missiles on Ukrainian soil striking into the Russian depth. The Flamingo, the FP5, is a British-made missile using German guidance equipment, brought to their target by American and NATO intelligence. Without this, Ukraine would not have this weapon. Ukraine's other weapons are using corridors that go through Poland into the Baltics so they can strike, ... in the vicinity of Saint Petersburg or through Kazakhstan so they can strike into the strategic depth.

Ignorance, the root and stem of all evil

37,607 次观看 • 4 个月前

Want to use Depth Anything, but need metric depth rather than relative depth? Thrilled to introduce Prompt Depth Anything, a new paradigm for accurate metric depth estimation with up to 4K resolution. 👉Key Message: Depth foundation models like DA have already internalized rich geometric knowledge of the 3D world but lack a proper way to elicit it. Inspired by the success of prompting in LLMs, we propose prompting Depth Anything with metric cues to produce metric depth. This method proves to be very effective when using a low-cost lidar (e.g., iPhone's LiDAR), which is widely available, as prompts. We believe the prompt can generalize to other forms as long as scale information is provided. Prompt Depth Anything offers 1⃣A series of models for iPhone lidars. 2⃣4D reconstruction from monocular videos (captured with iPhone). 3⃣Improved generalization ability for robot manipulation, e.g. Training on cans but generalizing on glasses. 4⃣More detailed depth annotations for the ScanNet++ dataset. The first author is our excellent intern Haotong Lin. Paper: Huggingface: Project Page: Code:

Want to use Depth Anything, but need metric depth rather than relative depth? Thrilled to introduce Prompt Depth Anything, a new paradigm for accurate metric depth estimation with up to 4K resolution. 👉Key Message: Depth foundation models like DA have already internalized rich geometric knowledge of the 3D world but lack a proper way to elicit it. Inspired by the success of prompting in LLMs, we propose prompting Depth Anything with metric cues to produce metric depth. This method proves to be very effective when using a low-cost lidar (e.g., iPhone's LiDAR), which is widely available, as prompts. We believe the prompt can generalize to other forms as long as scale information is provided. Prompt Depth Anything offers 1⃣A series of models for iPhone lidars. 2⃣4D reconstruction from monocular videos (captured with iPhone). 3⃣Improved generalization ability for robot manipulation, e.g. Training on cans but generalizing on glasses. 4⃣More detailed depth annotations for the ScanNet++ dataset. The first author is our excellent intern Haotong Lin. Paper: Huggingface: Project Page: Code:

Bingyi Kang

67,750 次观看 • 1 年前

Jane Fonda: “This is the first time in the history of the US that an authoritarian regime has broken through. Worldwide historians who study these things all agree it has never happened so fast in any industrialized democracy. He is moving to consolidate very very fast which means right now we have to use the tools we have to stop it”

Jane Fonda: “This is the first time in the history of the US that an authoritarian regime has broken through. Worldwide historians who study these things all agree it has never happened so fast in any industrialized democracy. He is moving to consolidate very very fast which means right now we have to use the tools we have to stop it”

Marco Foster

831,592 次观看 • 6 个月前