Introducing SwitchLight: Be anywhere at any time.

We are witnessing rapid advancements in visual generation technologies. The competitive landscape results in the high “fidelity” of the visuals improving daily. Yet, the “controllability” over these visuals remains surprisingly limited. In our recent study, we explore ways to precisely control “light”, which is a fundamental component that determines the style and mood of the imagery.

TL;DR SwitchLight is a co-design approach that combines physics-driven architecture with a self-supervised pre-training framework, pushing the state-of-the-art relighting quality significantly. We detail our method in the following.

First, we design an architecture that combines the best aspects of lighting physics with the data-driven capabilities of deep learning. We derive appropriate “inductive biases” from lighting physics and incorporate them into the model design. Beginning with a basic, singular convolutional UNet, we enhance it through three design principles.

Modular Design: Our approach is structured into two key phases: inverse rendering, where we determine the material properties consistent with the physical reflectance model we've selected, followed by re-rendering under the desired target lighting conditions.

Physically Aligned Output: Our rendering process adheres to the Cook-Torrance reflectance model, the most widely used physically-based reflectance model across modern rendering engines. This ensures that the intermediate material properties are aligned with this advanced reflectance model - even without dense intermediate supervision for materials.

Modeling Complex Lighting Effects: To address the intricate features that lighting physics may not precisely capture, such as self-occlusions and subsurface scatterings, we employ an additional neural decoder to learn these "residual" nuances directly from data.

Second, we develop a pre-training framework that facilitates the implicit scaling of relighting data, which are inherently difficult due to the need for specialized devices like light stages. The pretraining-then-finetuning paradigm has become a de-facto in training large language models. This approach is similarly explored in the vision domain, primarily for recognition tasks, while its application to generation tasks remains rare.

Multi-Masked Autoencoder: Our findings suggest that the masked autoencoder (MAE) framework is effective in enabling the model to learn transferrable low-level features, particularly when considering our prior understanding that convnets are prone to cheating during MAE-based pre-training ( This led us to design the task to be more challenging by incorporating diverse and difficult mask types.

Pre-training Data: Furthermore, the model shows a preference for generic data (e.g., ImageNet-IM) over task-specific data (e.g., Human portraits-H) when provided with equal amounts of data, underscoring the significance of input diversity over specificity during pre-training.

🔍Discover more about SwitchLight on Arxiv: 💡Our latest model also supports high-quality depth extraction - You can now extract albedo, normal, depth, roughness, specular, and lighting - all from a single image. 🎥Moreover, the entire network is trained to ensure temporal consistency - You can relight your favorite videos. Stay tuned to @beeble_ai for more advanced features that will be unveiled.

"I SWEAR BOSS I AM SITTING IN MY OFFICE RIGHT NOW!" Oh man, that's gonna be interesting to explain lol

@BenLumenDigital

This is so awesome 👍

Incredible work!!

If you have an image and mask, would it be able to relight that area?

@xiazai_shipin

Amazing!!

wow D:

@ATOM_VISUAL Une vidéo prochaine là-dessus ?