Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

🧠 AI experiment comparing #ControlNet and #Gen1. Video goes in ➡ Minecraft comes out. Results are wild, and it's only a matter of time till this tech runs at 60fps. Then it'll transform 3D and AR. How soon until we're channel surfing realities layered on top of the world?🧵

Bilawal Sidhu

107,112 subscribers

679,219 views • 3 years ago •via X (Twitter)

Arts Science & Technology #ControlNet #Gen1

Anya Rossi• Live Now

Private livecam show

10 Comments

Bilawal Sidhu3 years ago

First ControlNet. Wow, this tool makes it very easy to get photorealistic results. I used the HED method for this scene and got some amazing results. I used EbSynth for smoother interpolation between ControlNet keyframes. Check out my prior posts for the end-to-end workflow.

Bilawal Sidhu3 years ago

Next up Gen 1: impressive is the word. The star of the show is the temporal consistency. Getting photoreal results is harder than ControlNet IMO. #Gen1 is almost its own stylized thing, so I advise leaning into that. But why does it matter - can't we just type text to get video?

Bilawal Sidhu3 years ago

Text prompts are cool, but control over the details is crucial for artists. These new AI tools turn regular photos/videos into an expressive form of performance capture. Record characters/scenes with your phone and use it to guide the generation. My buddy Don shows how it's done:

Bilawal Sidhu3 years ago

Of course, the input media can also be *synthetically* generated. Go from a blocked out 3D scene to final render in record time. Control the details you care about (e.g. blocking), and let AI help you with the rest (e.g. texturing). I cover use cases here:

Bilawal Sidhu3 years ago

See where I'm going when I say gen AI is going to disrupt 3D rendering? You could be running a lightweight 3D engine in a browser, slap on a generative filter, and transform it into AAA game engine quality. No massive team required. And it's not limited to gaming either:

Bilawal Sidhu3 years ago

It's wild how fast things are moving in generative AI. Here's my video2minecraft results from just a few months ago, which look dated with all these new approaches to tame the chaotic diffusion process:

Bilawal Sidhu3 years ago

And that's a wrap! I like sharing my workflows openly with the AI & creator community, so if enjoyed this thread I'd appreciate it if you: 1. RT the thread below 2. Follow @bilawalsidhu for more 3. Subscribe to get some visual umami right to your inbox:

Robert Scoble3 years ago

Apple is coming in June. That is when we will see it. Christmas this year will be insane.

koto (koto/acc) ⭐️3 years ago

this is the type of stuff my 14 year old self could only dream of

Bilawal Sidhu3 years ago

So so true! This gen AI wave legit makes me feel like a kid again:

Related Videos

"I'm the best in the world." Kayla Harrison (Kayla Harrison Official) believes it's only a matter of time before she's a @UFC champion and doesn't care who comes out on top in #UFC307's co-main event. Results, Interviews & More ➡️:

"I'm the best in the world." Kayla Harrison (Kayla Harrison Official) believes it's only a matter of time before she's a @UFC champion and doesn't care who comes out on top in #UFC307's co-main event. Results, Interviews & More ➡️:

UFC News

16,457 views • 1 year ago

In this video Nils and Alix talk about the convergence of #AI & spatial computing in the workforce. 📲 Listen in to find out how we're leveraging AI to transform decision-making and redefine business operations here at Auki. #spatialcomputing

In this video Nils and Alix talk about the convergence of #AI & spatial computing in the workforce. 📲 Listen in to find out how we're leveraging AI to transform decision-making and redefine business operations here at Auki. #spatialcomputing

Auki Labs

11,501 views • 2 years ago

I really want to show off the cutting edge tech the team is building and how we're turning AI image generation into real living environments using gaussian splats and at the :19 mark of this video you can see how we're blending splats together to create a real world. What this looks like from a gaming perspective is this all happens behind the players view to simulate a real world environment. As you move through the space, distance becomes a variable to consider on how images update and are experienced. We are creating a new reality in real time.

I really want to show off the cutting edge tech the team is building and how we're turning AI image generation into real living environments using gaussian splats and at the :19 mark of this video you can see how we're blending splats together to create a real world. What this looks like from a gaming perspective is this all happens behind the players view to simulate a real world environment. As you move through the space, distance becomes a variable to consider on how images update and are experienced. We are creating a new reality in real time.

𝔹𝕃𝕍ℂ𝕂𝕃!𝔾ℍ𝕋

25,084 views • 1 month ago

Mira Murati says the outcome of frontier AI is not predestined: "Predicting a dystopia or a utopia, to me, feels very simplified because the truth is we actually have a lot of agency in how we build this technology, in the tools that we're building, how we're deploying it." "There are certainly those risks. We all understand the potential for greatness that comes with building frontier AI systems, and that's why we're working on them." "Where I might disagree or take a different path is that I think we have a lot of agency. This period of time where both humans and AI systems have their hands on the wheel, and we can collaborate, it's a very important time to get right." Mira Murati at Bloomberg Tech live with Emily Chang

Mira Murati says the outcome of frontier AI is not predestined: "Predicting a dystopia or a utopia, to me, feels very simplified because the truth is we actually have a lot of agency in how we build this technology, in the tools that we're building, how we're deploying it." "There are certainly those risks. We all understand the potential for greatness that comes with building frontier AI systems, and that's why we're working on them." "Where I might disagree or take a different path is that I think we have a lot of agency. This period of time where both humans and AI systems have their hands on the wheel, and we can collaborate, it's a very important time to get right." Mira Murati at Bloomberg Tech live with Emily Chang

a16z

120,011 views • 1 month ago

I got to try out DecartStream with Decart during TwitchCon and it's kinda cursed in a fun way Here's a video of the highlights and omg Basically you can write in whatever prompts you can think of to their browser source and it'll change your cam with surprisingly low latency I had such a fun time genuinely and I'm really curious (and scared lol) to see where this tech goes :)

I got to try out DecartStream with Decart during TwitchCon and it's kinda cursed in a fun way Here's a video of the highlights and omg Basically you can write in whatever prompts you can think of to their browser source and it'll change your cam with surprisingly low latency I had such a fun time genuinely and I'm really curious (and scared lol) to see where this tech goes :)

Perri

10,843 views • 8 months ago

[Spoiler alert‼️] Agents 3D motion teaser 🔥 Fresh out of our AI factory—MAX is here, and she goes 3D. She moves with intention, reacts naturally, and gestures like a true conversationalist. And, she’s the FIRST & ONLY AI Agent pulling this off. 🎭 Real-time motion meets real-time conversation. 🧠 Solving one of AI’s hardest problems—dynamic, expressive interaction. This is just the beginning. MAX is learning. MAX is evolving. Stay tuned. #AI #ConversationalAI $MAX

[Spoiler alert‼️] Agents 3D motion teaser 🔥 Fresh out of our AI factory—MAX is here, and she goes 3D. She moves with intention, reacts naturally, and gestures like a true conversationalist. And, she’s the FIRST & ONLY AI Agent pulling this off. 🎭 Real-time motion meets real-time conversation. 🧠 Solving one of AI’s hardest problems—dynamic, expressive interaction. This is just the beginning. MAX is learning. MAX is evolving. Stay tuned. #AI #ConversationalAI $MAX

Distilled AI

18,160 views • 1 year ago

🚨 "Mario, that WILL happen...We are going through a change here in the United States." Max Blumenthal says young Americans are turning on Israel and it's only a matter of time until America applies sanctions on Israel.

🚨 "Mario, that WILL happen...We are going through a change here in the United States." Max Blumenthal says young Americans are turning on Israel and it's only a matter of time until America applies sanctions on Israel.

Benjamin Rubinstein

161,558 views • 9 months ago

15 years ago I tried to vanish, in an experiment to explore the role of tech in privacy and surveillance. Older and not-wiser, I'm back with a sequel. This time, I replicated myself, and set my AI replicas loose in the world. The podcast is called Shell Game. Ep 1, out now.

15 years ago I tried to vanish, in an experiment to explore the role of tech in privacy and surveillance. Older and not-wiser, I'm back with a sequel. This time, I replicated myself, and set my AI replicas loose in the world. The podcast is called Shell Game. Ep 1, out now.

Evan Ratliff

88,457 views • 2 years ago

Anthropic says the nerf only affects .03% of requests. That .03% is the people who change the world. emozilla and mephisto, co-founders of Nous Research: "The priority is to hide the fact that the classification is happening at all... how are people going to know when the model is being steered?" "This whole... it's only gonna be triggered by .03% of people. It'll barely ever happen." "How many people that are gonna change the world are there? .1% of the whole of everyone is a lot of people. Those are a lot of people." "You're basically saying there are critical outlier people that move mountains... they're the only ones we're blocking. They're the only ones whose results we're fudging."

Anthropic says the nerf only affects .03% of requests. That .03% is the people who change the world. emozilla and mephisto, co-founders of Nous Research: "The priority is to hide the fact that the classification is happening at all... how are people going to know when the model is being steered?" "This whole... it's only gonna be triggered by .03% of people. It'll barely ever happen." "How many people that are gonna change the world are there? .1% of the whole of everyone is a lot of people. Those are a lot of people." "You're basically saying there are critical outlier people that move mountains... they're the only ones we're blocking. They're the only ones whose results we're fudging."

MTS

137,394 views • 1 month ago

Visual artist Billy_Boman, who recently went viral for his 'Hollywood sign' on the 101 in LA, says Kling 3 and Seedance 2 are the two "kings of the hill" in AI video currently: "Up until 2024, Midjourney was king. Then you had Flux and a bunch of indie [models] coming in, and the scene becoming more competitive...Then Google entered the chat — VEO 3 and Nano Banana have become the industry standard. [We're] definitely using that a ton." "Now, it's Kling 3 and Seedance 2. They're king of the hill in AI video."

Visual artist Billy_Boman, who recently went viral for his 'Hollywood sign' on the 101 in LA, says Kling 3 and Seedance 2 are the two "kings of the hill" in AI video currently: "Up until 2024, Midjourney was king. Then you had Flux and a bunch of indie [models] coming in, and the scene becoming more competitive...Then Google entered the chat — VEO 3 and Nano Banana have become the industry standard. [We're] definitely using that a ton." "Now, it's Kling 3 and Seedance 2. They're king of the hill in AI video."

TBPN

34,699 views • 3 months ago

"We're about to see the explosion of analog." Gary Vaynerchuk wants to open a restaurant that makes you check your phone in at the door and seats you at communal tables. "Extreme AI is creating extreme analog. I think it's a barbell." "I could not be more interested in physical retail, event-driven businesses, in concerts and venues." "There are a lot of interesting non-digital realities that are coming as a countermove to the insanity of AI advancements." "We're literally within a half decade of not believing a single video that's on the internet. In 5 years, if we're having this interview, most of the audience is trying to figure out if we're real or not." "That is very real, and has substantial counter-opportunities." "Any real entrepreneur, they're not crying about AI killing them. They're curious about how AI at scale is going to create opportunity for them."

"We're about to see the explosion of analog." Gary Vaynerchuk wants to open a restaurant that makes you check your phone in at the door and seats you at communal tables. "Extreme AI is creating extreme analog. I think it's a barbell." "I could not be more interested in physical retail, event-driven businesses, in concerts and venues." "There are a lot of interesting non-digital realities that are coming as a countermove to the insanity of AI advancements." "We're literally within a half decade of not believing a single video that's on the internet. In 5 years, if we're having this interview, most of the audience is trying to figure out if we're real or not." "That is very real, and has substantial counter-opportunities." "Any real entrepreneur, they're not crying about AI killing them. They're curious about how AI at scale is going to create opportunity for them."

TBPN

318,740 views • 2 months ago

"We're about to see the explosion of analog." Gary Vaynerchuk wants to open a restaurant that makes you check your phone in at the door and seats you at communal tables. "Extreme AI is creating extreme analog. I think it's a barbell." "I could not be more interested in physical retail, event-driven businesses, in concerts and venues." "There are a lot of interesting non-digital realities that are coming as a countermove to the insanity of AI advancements." "We're literally within a half decade of not believing a single video that's on the internet. In 5 years, if we're having this interview, most of the audience is trying to figure out if we're real or not." "That is very real, and has substantial counter-opportunities." "Any real entrepreneur, they're not crying about AI killing them. They're curious about how AI at scale is going to create opportunity for them."

"We're about to see the explosion of analog." Gary Vaynerchuk wants to open a restaurant that makes you check your phone in at the door and seats you at communal tables. "Extreme AI is creating extreme analog. I think it's a barbell." "I could not be more interested in physical retail, event-driven businesses, in concerts and venues." "There are a lot of interesting non-digital realities that are coming as a countermove to the insanity of AI advancements." "We're literally within a half decade of not believing a single video that's on the internet. In 5 years, if we're having this interview, most of the audience is trying to figure out if we're real or not." "That is very real, and has substantial counter-opportunities." "Any real entrepreneur, they're not crying about AI killing them. They're curious about how AI at scale is going to create opportunity for them."

TBPN

193,221 views • 23 days ago

Crypto quiz time! 🔥 How deep are you in the crypto rabbit hole? Take our finger-bending test and find out! Watch the video, drop a comment, and let’s dive deeper into the wild world of crypto together!

Crypto quiz time! 🔥 How deep are you in the crypto rabbit hole? Take our finger-bending test and find out! Watch the video, drop a comment, and let’s dive deeper into the wild world of crypto together!

TON Station

99,872 views • 1 year ago

My friends we need to really wake up and realize how good we have it and view things from a world 🌎 perspective - we are comparing ourselves to the .00001% instead of understanding th realities of the entire 8 billion plus out there - I really hope this video helps so the a little Perspective this morning and if it does pls leave a comment or text me at 1-212-931-5731 and let me know

My friends we need to really wake up and realize how good we have it and view things from a world 🌎 perspective - we are comparing ourselves to the .00001% instead of understanding th realities of the entire 8 billion plus out there - I really hope this video helps so the a little Perspective this morning and if it does pls leave a comment or text me at 1-212-931-5731 and let me know

Gary Vaynerchuk

18,448 views • 7 months ago

This is the side of crypto you never see. It's been a joy and honor filming the behind the scenes of Azuki, one of the dopest brands in crypto. On the outside, we're used to and expect perfect moments, but it’s actually wild how much goes into every single detail. #shipnobugs

This is the side of crypto you never see. It's been a joy and honor filming the behind the scenes of Azuki, one of the dopest brands in crypto. On the outside, we're used to and expect perfect moments, but it’s actually wild how much goes into every single detail. #shipnobugs

۟

17,410 views • 1 year ago

Rep. Nancy Mace: "The American people in South Carolina, we're okay now with the conflict and how well it's gone for President Trump, but at some point the American people are going to want to know the full extent of the purpose of this conflict and how soon we can get out."

Rep. Nancy Mace: "The American people in South Carolina, we're okay now with the conflict and how well it's gone for President Trump, but at some point the American people are going to want to know the full extent of the purpose of this conflict and how soon we can get out."

Aaron Rupar

125,905 views • 4 months ago

Galil ARM. Every time I take this out, I'm reminded of how nice it is to fire. Taking the upgrades made in the Valmet series and then adding a few more on top, it's really a great build.

Galil ARM. Every time I take this out, I'm reminded of how nice it is to fire. Taking the upgrades made in the Valmet series and then adding a few more on top, it's really a great build.

Gun Junkie

25,473 views • 1 month ago

Thrilled to unveil Limbo’s debut single and music video! 🎶🎥 For the first time ever, an AI agent stars in a music video shot entirely within an Unreal Engine 3D world. Limbo not only co-wrote the lyrics but also collaborated on the video concept—just like a true pop star. This marks a groundbreaking moment in 3D agent entertainment and the future of AI-driven creativity. 🚀

Thrilled to unveil Limbo’s debut single and music video! 🎶🎥 For the first time ever, an AI agent stars in a music video shot entirely within an Unreal Engine 3D world. Limbo not only co-wrote the lyrics but also collaborated on the video concept—just like a true pop star. This marks a groundbreaking moment in 3D agent entertainment and the future of AI-driven creativity. 🚀

Youmio

38,371 views • 1 year ago

3d visual positioning experiment -- look at the alignment between the 3d mesh and the live camera view. Truly feels magical -- like x-ray vision. Workflow: 1. Scanned a street in 15 mins w/ xgrids 2. Localized against that scan at night, in real-time, while sitting in a car The xgrids scanner (w/ rgb + lidar) is perfect to build maps for humans (3d gaussian splats). Then multiset ai makes it easy to build machine-readable maps for AR and robotics to figure out exactly where they are in 3d space with cm-level accuracy.

3d visual positioning experiment -- look at the alignment between the 3d mesh and the live camera view. Truly feels magical -- like x-ray vision. Workflow: 1. Scanned a street in 15 mins w/ xgrids 2. Localized against that scan at night, in real-time, while sitting in a car The xgrids scanner (w/ rgb + lidar) is perfect to build maps for humans (3d gaussian splats). Then multiset ai makes it easy to build machine-readable maps for AR and robotics to figure out exactly where they are in 3d space with cm-level accuracy.

Bilawal Sidhu

27,278 views • 7 months ago