Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Sharing something exciting we've been working on as a Thanksgiving gift: Diffusion Self-Distillation (DSD), which redefines zero-shot customized image generation using FLUX. DSD is like DreamBooth, but zero-shot/training-free. It works across any input subject and desired context—character consistency, item/asset adaptation, scene relighting, and more. It even enables the creation... show more

Prime (Shengqu) Cai

2,201 subscribers

60,760 Aufrufe • vor 1 Jahr •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

45 Kommentare

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

HF discussion page:

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

A simple 9-panel comic made with our model --- it took me less than 10 minutes

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

And another one

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Character Adaptation

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Item/Asset Adaptation --- could be useful for commercial Ads!

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

InstructPix2Pix-type Instruction Prompting (we can extract different subjects from an input image too!)

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Scene Relighting

Profilbild von Amogh Vaishampayan

Amogh Vaishampayanvor 1 Jahr

Will you be releasing the code? My team builds Dashtoon Studio to make comics with AI and we distribute and monetise them through the Dashtoon mobile app. Would love to try it out and give you feedback by trying it out in our large volume production.

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

yes, it will be open sourced in the future!

Profilbild von Hrmkmdtv

Hrmkmdtvvor 1 Jahr

Great developments

Profilbild von Nir Zabari‎

Nir Zabari‎vor 1 Jahr

Really cool work! 💥

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Thx 😬

Profilbild von Iván de Prado

Iván de Pradovor 1 Jahr

This is a great idea. Congratulations.

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Thx! 😬

Profilbild von Junlin Han

Junlin Hanvor 1 Jahr

Results are looking super cool! 🎉!

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Thanks Junlin! 😃

Profilbild von Sophia He

Sophia Hevor 1 Jahr

Awesome work!

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Thx! 😬

Profilbild von Andrei Filatov

Andrei Filatovvor 1 Jahr

Awesome)

Profilbild von Stelfie the Time Traveller

Stelfie the Time Travellervor 1 Jahr

great results! are you going to open it to test?

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Thanks and yes, I'm training a more stable Grande version and as soon as that's finished, I'll try to open up a Gradio demo/merge to ComfyUI

Profilbild von Stelfie the Time Traveller

Stelfie the Time Travellervor 1 Jahr

that would be awesome, if you need testers let me know pls!

Profilbild von Mohamad Shahbazi

Mohamad Shahbazivor 1 Jahr

Wow cool results Shengqu!

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Thanks Mohamad! 😬

Profilbild von UTSBH

UTSBHvor 1 Jahr

Is the model available in HF?

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Not yet. I am training a steadier "Grande" version, and will open source as soon as that's finished.

Profilbild von s______-O🏳️‍🌈

s______-O🏳️‍🌈vor 1 Jahr

Can you support SD3.5 please?

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

I think this strongly depends on how the ecosystem of SD3.5 goes --- personally I'd love to try the method on a non-distilled vanilla model.

Profilbild von Jin_Jin

Jin_Jinvor 1 Jahr

Amazing work

Profilbild von Valboa

Valboavor 1 Jahr

@_akhaliq $Trias for the win🚀🚀🚀

Profilbild von Jeff Liang

Jeff Liangvor 1 Jahr

Super cool! A smart way to curate diverse paired data. I am curious what is the success rate.

Profilbild von Next AI Tool

Next AI Toolvor 1 Jahr

Congratulations on your work and happy Thanksgiving! We've added it to The Next AI Tool.

Profilbild von 伊果均

伊果均vor 1 Jahr

Awesome, Thank you.

Profilbild von Parzival - ∞/89

Parzival - ∞/89vor 1 Jahr

Amazing work.

Profilbild von Ben

Benvor 1 Jahr

What is this sorcery?! That’s amazing

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Will write about that a bit later! But in short, we distill FLUX into a conditional two-frame video generation model, using its own outputs --- which in fact already contain large portions of identity-preserving contents!

Profilbild von Ben

Benvor 1 Jahr

Thanks for sharing the insights! I’d love to read more about it :)

Profilbild von Parzival - ∞/89

Parzival - ∞/89vor 1 Jahr

Do you think there will be the possibility of the same thing but for multiple characters in a scene in the future with a similar approach but many images?

Profilbild von Cryptexis

Cryptexisvor 1 Jahr

excellent work! Tried to reach out to you via email. Will try it again here :) It seems after generating grids you end-up with 4 256x256 images. Do you use upscaling before fine-tuning the parallel processing architecture? If yes, which one ? :)

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

We generate 1024×1024 grids so the pairs will be 512×512 --- thus all images here are also 512×512. However, we find the framework work well to higher resolutions, since the approach is scalable to what the base model can do. If we directly generative pairs instead of grids, the resolution can easily go up to 960×960, etc.

Profilbild von Cryptexis

Cryptexisvor 1 Jahr

thanks. Makes sense

Profilbild von Zhi Liao

Zhi Liaovor 1 Jahr

Amazing work

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

Thx! 😬

Profilbild von Crystalwizard

Crystalwizardvor 1 Jahr

so much promise in this now - if you want it used, create nodes for @ComfyUI

Profilbild von Prime (Shengqu) Cai

Prime (Shengqu) Caivor 1 Jahr

@ComfyUI on my schedule!

Ähnliche Videos

StyleDrop: Text-to-Image Generation in Any Style introduce StyleDrop, a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. The proposed method is extremely versatile and captures nuances and details of a user-provided style, such as color schemes, shading, design patterns, and local and global effects. It efficiently learns a new style by fine-tuning very few trainable parameters (less than 1% of total model parameters) and improving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user supplies only a single image that specifies the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop implemented on Muse convincingly outperforms other methods, including DreamBooth and textual inversion on Imagen or Stable Diffusion. paper page:

StyleDrop: Text-to-Image Generation in Any Style introduce StyleDrop, a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. The proposed method is extremely versatile and captures nuances and details of a user-provided style, such as color schemes, shading, design patterns, and local and global effects. It efficiently learns a new style by fine-tuning very few trainable parameters (less than 1% of total model parameters) and improving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user supplies only a single image that specifies the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop implemented on Muse convincingly outperforms other methods, including DreamBooth and textual inversion on Imagen or Stable Diffusion. paper page:

AK

56,397 Aufrufe • vor 3 Jahren

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior paper page: present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation.

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior paper page: present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation.

AK

161,530 Aufrufe • vor 2 Jahren

I couldn't believe it. One product image = a full video ad with Wizstar. No script. No shoot. No editor. Just the image, one sentence of instruction, and Wizstar did everything else. It analyzes your product automatically, builds the scene around it, and generates a complete marketing video. Character stays consistent across every shot. Product details stay accurate. Audio and video sync perfectly. It powers multi-model invocation to match any style you need, and Seedance 2.0 supports face input for consistent character generation. But wait, it gets better. You can also upload any viral video from TikTok or Amazon. Then it will recreate that EXACT winning formula for YOUR product. No more guessing what works. Zero creative trial and error. Make a video today, start seeing sales in 7 days. New users get free credits on signup. First subscription is only $19 and includes a 30-second E-commerce Agent experience. Try it here: Who's making their first ad tonight? 👇 #Wizstar #AIVideo #EcommerceMarketing

I couldn't believe it. One product image = a full video ad with Wizstar. No script. No shoot. No editor. Just the image, one sentence of instruction, and Wizstar did everything else. It analyzes your product automatically, builds the scene around it, and generates a complete marketing video. Character stays consistent across every shot. Product details stay accurate. Audio and video sync perfectly. It powers multi-model invocation to match any style you need, and Seedance 2.0 supports face input for consistent character generation. But wait, it gets better. You can also upload any viral video from TikTok or Amazon. Then it will recreate that EXACT winning formula for YOUR product. No more guessing what works. Zero creative trial and error. Make a video today, start seeing sales in 7 days. New users get free credits on signup. First subscription is only $19 and includes a 30-second E-commerce Agent experience. Try it here: Who's making their first ad tonight? 👇 #Wizstar #AIVideo #EcommerceMarketing

Pradeep Pandey

59,917 Aufrufe • vor 3 Monaten

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.

AK

40,490 Aufrufe • vor 2 Jahren

A robot hand grasp over 500 totally new objects without fail? Zero-shot, single-view & super reliable ⬇️ + Paper Grasping random objects is hard for robots, especially when shapes, weights, and materials vary. RobustDexGrasp solves this with a smart new way of seeing and controlling the hand, leading to near-perfect grasping, even in noisy or cluttered scenes. Thank you for sharing, Hui Zhang 🙏 Follow him!! What makes it special ✅ Grabs 500+ unseen objects with 94.6% success using only single-view input ✅ Learns local shapes, not full geometry, for better generalization ✅ Trained with just 35 objects in sim but works in the real world with hundreds more ✅ Adapts to noise, unexpected forces, and even plays chess with VLM planning It shows that smart sensing and adaptive control can take dexterous grasping to the next level. Project: Paper:

A robot hand grasp over 500 totally new objects without fail? Zero-shot, single-view & super reliable ⬇️ + Paper Grasping random objects is hard for robots, especially when shapes, weights, and materials vary. RobustDexGrasp solves this with a smart new way of seeing and controlling the hand, leading to near-perfect grasping, even in noisy or cluttered scenes. Thank you for sharing, Hui Zhang 🙏 Follow him!! What makes it special ✅ Grabs 500+ unseen objects with 94.6% success using only single-view input ✅ Learns local shapes, not full geometry, for better generalization ✅ Trained with just 35 objects in sim but works in the real world with hundreds more ✅ Adapts to noise, unexpected forces, and even plays chess with VLM planning It shows that smart sensing and adaptive control can take dexterous grasping to the next level. Project: Paper:

Ilir Aliu

37,980 Aufrufe • vor 1 Jahr

We've officially released and open-sourced HunyuanImage 2.1, our latest text-to-image model. The new model delivers on our commitment to balancing performance and quality. With native 2K image generation, HunyuanImage 2.1 is an advanced open-source text-to-image model.🎨 ✨ New in 2.1: 🔹Advanced Semantics: Supports ultra-long and complex prompts of up to 1000 tokens, and precisely controls the generation of multiple subjects in a single image. 🔹Precise Chinese and English Text Rendering with seamless image–text integration: The model naturally integrates text into images, making it suitable for a wide range of applications such as product covers, illustrations, and poster design to meet the needs of various fields. 🔹Rich Styles and High Aesthetic: Capable of generating images in various styles—including photorealistic portraits, comics, and vinyl figures—it delivers outstanding visual appeal and artistic quality. 🔹High-Quality Generation: Efficiently produces ultra-high-definition (2K) images in the same time other models take to generate a 1K image. HunyuanImage 2.1 uses two text encoders: a multimodal large language model (MLLM) to improve the model's image and text alignment capabilities, and a multi-language character-aware encoder to improve text rendering capabilities. The model is a single- and double-stream diffusion transformer with 17B parameters. We've also open-sourced the weights of the the accelerated version with meanflow which reduces inference steps from 100 to just 8, and PromptEnhancer, the first industrial-grade rewriting model that enhances your prompts for more nuanced and expressive image generation. Now, creators turn complex ideas—like posters with slogans or multi-panel comics—into visuals faster than ever. We’re just getting started. Stay tuned for our native multimodal image generation model coming soon. 🌐Website: 🔗Github: 🤗Hugging Face: ✨Hugging Face Demo:

We've officially released and open-sourced HunyuanImage 2.1, our latest text-to-image model. The new model delivers on our commitment to balancing performance and quality. With native 2K image generation, HunyuanImage 2.1 is an advanced open-source text-to-image model.🎨 ✨ New in 2.1: 🔹Advanced Semantics: Supports ultra-long and complex prompts of up to 1000 tokens, and precisely controls the generation of multiple subjects in a single image. 🔹Precise Chinese and English Text Rendering with seamless image–text integration: The model naturally integrates text into images, making it suitable for a wide range of applications such as product covers, illustrations, and poster design to meet the needs of various fields. 🔹Rich Styles and High Aesthetic: Capable of generating images in various styles—including photorealistic portraits, comics, and vinyl figures—it delivers outstanding visual appeal and artistic quality. 🔹High-Quality Generation: Efficiently produces ultra-high-definition (2K) images in the same time other models take to generate a 1K image. HunyuanImage 2.1 uses two text encoders: a multimodal large language model (MLLM) to improve the model's image and text alignment capabilities, and a multi-language character-aware encoder to improve text rendering capabilities. The model is a single- and double-stream diffusion transformer with 17B parameters. We've also open-sourced the weights of the the accelerated version with meanflow which reduces inference steps from 100 to just 8, and PromptEnhancer, the first industrial-grade rewriting model that enhances your prompts for more nuanced and expressive image generation. Now, creators turn complex ideas—like posters with slogans or multi-panel comics—into visuals faster than ever. We’re just getting started. Stay tuned for our native multimodal image generation model coming soon. 🌐Website: 🔗Github: 🤗Hugging Face: ✨Hugging Face Demo:

Tencent Hy

89,392 Aufrufe • vor 1 Jahr

Tencent presents GameGen-O Open-world Video Game Generation We introduce GameGen-O, the first diffusion transformer model tailored for the generation of open-world video games. This model facilitates high-quality, open-domain generation by simulating a wide array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, thus allowing for the gameplay simulation. The development of GameGen-O involves a comprehensive data collection and processing effort from scratch. We collect and build the first Open-World Video Game Dataset (OGameData), amassed extensive data from over a hundred of next-generation open-world games, employing a proprietary data pipeline for efficient sorting, scoring, filtering, and decoupled captioning. This robust and extensive OGameData forms the foundation of our model's training process. GameGen-O undergoes a two-stage training process, consisting of foundation model pretraining and instruction tuning. In the first phase, the model is pre-trained on the OGameData via the text-to-video and video continuation, endowing GameGen-O with the capability for open-domain video game generation. In the second phase, the pre-trained model is frozen, and we fine-tuned using a trainable InstructNet, which enables the production of subsequent frames based on multimodal structural instructions. This whole training process imparts the model with the ability to generate and interactively control content. In summary, GameGen-O represents a notable initial step forward in the realm of open-world video game generation via generative models. It underscores the potential of generative models to serve as an alternative to rendering techniques, which can efficiently combine creative generation with interactive capabilities.

Tencent presents GameGen-O Open-world Video Game Generation We introduce GameGen-O, the first diffusion transformer model tailored for the generation of open-world video games. This model facilitates high-quality, open-domain generation by simulating a wide array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events. Additionally, it provides interactive controllability, thus allowing for the gameplay simulation. The development of GameGen-O involves a comprehensive data collection and processing effort from scratch. We collect and build the first Open-World Video Game Dataset (OGameData), amassed extensive data from over a hundred of next-generation open-world games, employing a proprietary data pipeline for efficient sorting, scoring, filtering, and decoupled captioning. This robust and extensive OGameData forms the foundation of our model's training process. GameGen-O undergoes a two-stage training process, consisting of foundation model pretraining and instruction tuning. In the first phase, the model is pre-trained on the OGameData via the text-to-video and video continuation, endowing GameGen-O with the capability for open-domain video game generation. In the second phase, the pre-trained model is frozen, and we fine-tuned using a trainable InstructNet, which enables the production of subsequent frames based on multimodal structural instructions. This whole training process imparts the model with the ability to generate and interactively control content. In summary, GameGen-O represents a notable initial step forward in the realm of open-world video game generation via generative models. It underscores the potential of generative models to serve as an alternative to rendering techniques, which can efficiently combine creative generation with interactive capabilities.

AK

367,249 Aufrufe • vor 2 Jahren

Okay, this one is the Grok Imagine version. But here’s something you need to note: I uploaded the character sheet I made with Nano Banana Pro as the base image. So it feels like this fake “Omni reference” kind of style, but it works 😄 Somehow, it’s much more consistent than uploading a still image of your scene with your character, because there’s a chance your character will morph a bit. By using your character sheet as the base model, it helps maintain consistency. The only problem is that when you do the img2vid prompt, you need to retell the setting, the action, the vibe and so on. But for me, I kind of like that approach, LOL 😆 Like I said, I’m getting lazy about animating images one by one. With this approach, I can just focus on dividing the shots, like shot 1: blabla, shot 2: bliblibli, shot 3: blublu, etc. Now for the comparison: In comparison with the Kling 3.0 Omni version, I feel that Grok Imagine is much more dynamic. But in terms of the character performance, it feels a bit over the top. It’s like the AI characters are acting it out 😁 Meanwhile, Kling 3.0 Omni feels much more realistic in terms of the acting and a bit more atmospheric. I honestly love both. If I want to fool around, I guess I’ll use Grok Imagine. But if I want more subtle realism and a candid feel, I’ll go with Kling 3.0.

Okay, this one is the Grok Imagine version. But here’s something you need to note: I uploaded the character sheet I made with Nano Banana Pro as the base image. So it feels like this fake “Omni reference” kind of style, but it works 😄 Somehow, it’s much more consistent than uploading a still image of your scene with your character, because there’s a chance your character will morph a bit. By using your character sheet as the base model, it helps maintain consistency. The only problem is that when you do the img2vid prompt, you need to retell the setting, the action, the vibe and so on. But for me, I kind of like that approach, LOL 😆 Like I said, I’m getting lazy about animating images one by one. With this approach, I can just focus on dividing the shots, like shot 1: blabla, shot 2: bliblibli, shot 3: blublu, etc. Now for the comparison: In comparison with the Kling 3.0 Omni version, I feel that Grok Imagine is much more dynamic. But in terms of the character performance, it feels a bit over the top. It’s like the AI characters are acting it out 😁 Meanwhile, Kling 3.0 Omni feels much more realistic in terms of the acting and a bit more atmospheric. I honestly love both. If I want to fool around, I guess I’ll use Grok Imagine. But if I want more subtle realism and a candid feel, I’ll go with Kling 3.0.

MXVDXN // DAN

16,961 Aufrufe • vor 7 Monaten

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal

Adithya Murali

24,347 Aufrufe • vor 1 Jahr

TEDRA Text-based Editing of Dynamic and Photoreal Actors discuss: Over the past years, significant progress has been made in creating photorealistic and drivable 3D avatars solely from videos of real humans. However, a core remaining challenge is the fine-grained and user-friendly editing of clothing styles by means of textual descriptions. To this end, we present TEDRA, the first method allowing text-based edits of an avatar, which maintains the avatar's high fidelity, space-time coherency, as well as dynamics, and enables skeletal pose and view control. We begin by training a model to create a controllable and high-fidelity digital replica of the real actor. Next, we personalize a pretrained generative diffusion model by fine-tuning it on various frames of the real character captured from different camera angles, ensuring the digital representation faithfully captures the dynamics and movements of the real person. This two-stage process lays the foundation for our approach to dynamic human avatar editing. Utilizing this personalized diffusion model, we modify the dynamic avatar based on a provided text prompt using our Personalized Normal Aligned Score Distillation Sampling (PNA-SDS) within a model-based guidance framework. Additionally, we propose a time step annealing strategy to ensure high-quality edits. Our results demonstrate a clear improvement over prior work in functionality and visual quality.

AK

17,302 Aufrufe • vor 2 Jahren

fable, I want you to grab a MetaHuman file off of Unreal Engine Marketplace or Fab Marketplace or something like that. There should be free ones. Find a character controller that works with that for web and threeJS. Specifically for flying, like Superman, generate a World Labs outdoor environment of your choosing and set up a threeJS project where we can fly around with a character like Superman, with really nice flying animations, camera animations, and camera zoom and shake effects when you're flying. Put some nice post-processing on it with some bloom and grain and stuff like that. Really just make this feel as polished as you can, where we get to fly around in a world where the flight feels similar to Superman. Do it in one shot and without any input from me. optimize it 2 prompts - NOT BAD!! Honestly could get this feeling/looking 10x better with probably 2 more prompts. I'd likely use a larger city GLB, populate the outskirts with World Labs gsplats, manually grab a better character to use, and add some objectives. But I wanted to see what it could do in 1 prompt (basically) Try it out 👇

fable, I want you to grab a MetaHuman file off of Unreal Engine Marketplace or Fab Marketplace or something like that. There should be free ones. Find a character controller that works with that for web and threeJS. Specifically for flying, like Superman, generate a World Labs outdoor environment of your choosing and set up a threeJS project where we can fly around with a character like Superman, with really nice flying animations, camera animations, and camera zoom and shake effects when you're flying. Put some nice post-processing on it with some bloom and grain and stuff like that. Really just make this feel as polished as you can, where we get to fly around in a world where the flight feels similar to Superman. Do it in one shot and without any input from me. optimize it 2 prompts - NOT BAD!! Honestly could get this feeling/looking 10x better with probably 2 more prompts. I'd likely use a larger city GLB, populate the outskirts with World Labs gsplats, manually grab a better character to use, and add some objectives. But I wanted to see what it could do in 1 prompt (basically) Try it out 👇

Nicholas | WithLore

88,652 Aufrufe • vor 3 Monaten

Gemini-1.5 Pro has its spotlight stolen today, and people are poking fun at Sora vs Google memes. Well, I think it's the biggest boost in LLM capability so far in 2024. v1.5's 10M token context (1) excels at retrieval; (2) generalizes zero-shot to extremely long instructions like full tutorials and codebases; and (3) works across modalities such as text, audio, and video. Here's a stunning example: v1.5 learns to translate from English to Kalamang purely in context, following a full linguistic manual at inference time. Kalamang is a language spoken by fewer than 200 speakers in western New Guinea. Gemini has never seen this language during training and is only provided with 500 pages of linguistic documentation, a dictionary, and ~400 parallel sentences in context. It basically acquires a sophisticated new skill in the neural activations, instead of gradient finetuning. I talked about the Myth of Context Length many times before: don't get too excited by claims of 1M or even 1B context tokens. LSTMs already achieved literally infinite context length 25 yrs ago! What truly matters is how well the model actually uses the context to solve real-world problems, and Gemini-1.5 has surpassed the SOTA with flying colors. The paper is also well-written with lots of solid quantitative analysis on in-context memorization and generalization. Paper: “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” Congrats to Jeff Dean Oriol Vinyals Sundar Pichai and team!

Gemini-1.5 Pro has its spotlight stolen today, and people are poking fun at Sora vs Google memes. Well, I think it's the biggest boost in LLM capability so far in 2024. v1.5's 10M token context (1) excels at retrieval; (2) generalizes zero-shot to extremely long instructions like full tutorials and codebases; and (3) works across modalities such as text, audio, and video. Here's a stunning example: v1.5 learns to translate from English to Kalamang purely in context, following a full linguistic manual at inference time. Kalamang is a language spoken by fewer than 200 speakers in western New Guinea. Gemini has never seen this language during training and is only provided with 500 pages of linguistic documentation, a dictionary, and ~400 parallel sentences in context. It basically acquires a sophisticated new skill in the neural activations, instead of gradient finetuning. I talked about the Myth of Context Length many times before: don't get too excited by claims of 1M or even 1B context tokens. LSTMs already achieved literally infinite context length 25 yrs ago! What truly matters is how well the model actually uses the context to solve real-world problems, and Gemini-1.5 has surpassed the SOTA with flying colors. The paper is also well-written with lots of solid quantitative analysis on in-context memorization and generalization. Paper: “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context” Congrats to Jeff Dean Oriol Vinyals Sundar Pichai and team!

Jim Fan

278,557 Aufrufe • vor 2 Jahren

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

The All-In Podcast

54,050 Aufrufe • vor 11 Tagen

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

AK

429,355 Aufrufe • vor 3 Jahren

Head of Nemotron Bryan Catanzaro says NVIDIA is sitting on internal "teacher" models that beat everything it has publicly shipped and he's trying to release these as well "This technology of using teacher models and multi-domain on-policy distillation has been really key for us in these ways." "Now, what does that mean? Well, it definitely means that inside NVIDIA we have a collection of teacher models that are individually better at different domains than any of the models that we've released." "And actually, I'm trying to release those teacher models as part of the Nemotron project as well -- because as you know, Nemotron is not just a model." "Nemotron is an open technology effort. It includes data sets, it includes techniques, all of our ideas about how to build a model, which we publish." "Of course it includes the models themselves, but I think it should also include collections of teacher models." "I'd imagine this could be a really great way of federating open technologies for AI across the ecosystem." _______ Wrote more on a prior Bryan interview here:

Head of Nemotron Bryan Catanzaro says NVIDIA is sitting on internal "teacher" models that beat everything it has publicly shipped and he's trying to release these as well "This technology of using teacher models and multi-domain on-policy distillation has been really key for us in these ways." "Now, what does that mean? Well, it definitely means that inside NVIDIA we have a collection of teacher models that are individually better at different domains than any of the models that we've released." "And actually, I'm trying to release those teacher models as part of the Nemotron project as well -- because as you know, Nemotron is not just a model." "Nemotron is an open technology effort. It includes data sets, it includes techniques, all of our ideas about how to build a model, which we publish." "Of course it includes the models themselves, but I think it should also include collections of teacher models." "I'd imagine this could be a really great way of federating open technologies for AI across the ecosystem." _______ Wrote more on a prior Bryan interview here:

Fireside Alpha

29,125 Aufrufe • vor 1 Monat

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:

New Course: Reinforcement Fine-Tuning LLMs with GRPO! Learn to use reinforcement learning to improve your LLM performance in this short course, built in collaboration with Predibase by Rubrik, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead. Reasoning models have been one of the most important developments in LLMs. Reinforcement Fine-Tuning (RFT) uses rewards to encourage LLMs to find solutions to multi-step reasoning tasks such as solving math problems and debugging code - without needing pre-existing training examples like in traditional supervised fine-tuning. Group Relative Policy Optimization (GRPO) is a reinforcement fine-tuning algorithm gaining rapid adoption. Developed by the DeepSeek team and used to train the R1 reasoning model, GRPO uses reward functions that you can write in Python to assign rewards to model responses. It’s beneficial for tasks with verifiable outcomes and can work well even with fewer than 100 training examples. It can also significantly improve the reasoning ability of smaller LLMs, making applications faster and more cost effective. In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks. In detail, you’ll: - Learn when reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data. - Understand how GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO. - Frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time. - Design reward functions that power the reinforcement fine-tuning process. - Learn techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge. - Understand why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors. - Learn the four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence. - Launch reinforcement fine-tuning jobs using Predibase’s hosted training services. By the end of this course, you’ll be able to build and fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback. Please sign up here:

Andrew Ng

86,697 Aufrufe • vor 1 Jahr

The robot flipped a pancake nobody taught it! 🥞 Skild AI team assumed pancake flipping had to be somewhere in the training data. So they searched. Millions of hours of pre-training data. Nothing. S1 inferred the whole task from a single human demonstration. That's their new general robot model, built as an in-context learner from the ground up. Every new robot task today starts with days of teleoperation and a fine-tuning run on a specialist policy. S1 skips all of it. Much like a language model, it never updates its weights to learn a new task. The demonstration enters the context window, and the policy uses it to decide what to do next. → Ten-minute tasks it was never trained on, composed from primitives learned in pre-training: a new style of coffee, potting a plant, frying pancakes. → Soil and pots arrived at their office at 8:54 PM. The robot was running the task autonomously by 9:27 PM. → Slide objects away mid-reach, swap them, change the lighting, it still finishes. → The prompt waters a plant with a watering can, but only a cup is available. It uses the cup. It doesn't rigidly replay what it saw, but it recovers from its own errors, and sometimes executes with more precision than the demonstrator, when the human fumbles an egg and makes a mess, S1 performs the same step cleanly. The demonstration is a specification of the goal, and not a trajectory to copy. On unseen tasks after 100K hours of pre-training: language-prompted VLAs reach 9%. Their new model reaches 66%. It's already deploying with industrial partners, with a wider rollout over the coming months. Congrats Deepak Pathak and team behind this! 😮‍💨 🔗 Link to their latest blog: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

The robot flipped a pancake nobody taught it! 🥞 Skild AI team assumed pancake flipping had to be somewhere in the training data. So they searched. Millions of hours of pre-training data. Nothing. S1 inferred the whole task from a single human demonstration. That's their new general robot model, built as an in-context learner from the ground up. Every new robot task today starts with days of teleoperation and a fine-tuning run on a specialist policy. S1 skips all of it. Much like a language model, it never updates its weights to learn a new task. The demonstration enters the context window, and the policy uses it to decide what to do next. → Ten-minute tasks it was never trained on, composed from primitives learned in pre-training: a new style of coffee, potting a plant, frying pancakes. → Soil and pots arrived at their office at 8:54 PM. The robot was running the task autonomously by 9:27 PM. → Slide objects away mid-reach, swap them, change the lighting, it still finishes. → The prompt waters a plant with a watering can, but only a cup is available. It uses the cup. It doesn't rigidly replay what it saw, but it recovers from its own errors, and sometimes executes with more precision than the demonstrator, when the human fumbles an egg and makes a mess, S1 performs the same step cleanly. The demonstration is a specification of the goal, and not a trajectory to copy. On unseen tasks after 100K hours of pre-training: language-prompted VLAs reach 9%. Their new model reaches 66%. It's already deploying with industrial partners, with a wider rollout over the coming months. Congrats Deepak Pathak and team behind this! 😮‍💨 🔗 Link to their latest blog: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

11,406 Aufrufe • vor 29 Tagen

We've built 40+ AI agents and internal tools. The hardest part is Context Creation. AI runs playbooks and makes judgment calls for you. But without your company's context, you get slop. Context Creation means extracting the subject matter expertise and playbooks that live in people's heads, not in LLM training data, or even your tools. As forward deployed engineers (FDEs), we create context and turn it into code. We evaluate the business impact, how it aligns with the dev roadmap, and come up with creative solutions. We built The FDE Factory to replace ourselves. It drives AI adoption inside our clients' companies by running discovery sessions using prototypes to create context. Here's how it works: We put a prototype in front of a stakeholder. The stakeholder gives feedback via voice while they're using or reviewing it. Then our FDE Factory Agents builds in their expertise in minutes: > Context Agent reviews the codebase and feedback, extracts the requirements, and creates a spec > Scope Agent checks the spec against the development roadmap, validates it, and hands it off > Engineering Agent builds a new feature and wires the integration > QA Agent runs tests to prove to itself it works > PR merges, feature goes live, product updates itself in real time It's like the nontechnical stakeholder wrote the code without even knowing it. Coding agents are great at turning good development plans into code, and they're getting better at turning context into good development plans in collaboration with professional engineers. But nontechnical people are capped on what they can build without product people and engineers. The bridge that takes nontechnical people from vibe coding basic apps to building production AI tools that run on first party context is FDEs. Our new FDE Factory gives you the system to go from idea to production. Context Creation is the first and most important step in our FDE lifecycle, and we just automated it. Now clients get the right agents and tools built for them, customized to their unique business and encoded with their expertise. PS: If you're building AI agents within your company, reply "Playbook" and I'll DM you the entire FDE playbook we've run with 30+ companies. It covers finding high-impact AI use cases, building them, and deploying them across the org.

We've built 40+ AI agents and internal tools. The hardest part is Context Creation. AI runs playbooks and makes judgment calls for you. But without your company's context, you get slop. Context Creation means extracting the subject matter expertise and playbooks that live in people's heads, not in LLM training data, or even your tools. As forward deployed engineers (FDEs), we create context and turn it into code. We evaluate the business impact, how it aligns with the dev roadmap, and come up with creative solutions. We built The FDE Factory to replace ourselves. It drives AI adoption inside our clients' companies by running discovery sessions using prototypes to create context. Here's how it works: We put a prototype in front of a stakeholder. The stakeholder gives feedback via voice while they're using or reviewing it. Then our FDE Factory Agents builds in their expertise in minutes: > Context Agent reviews the codebase and feedback, extracts the requirements, and creates a spec > Scope Agent checks the spec against the development roadmap, validates it, and hands it off > Engineering Agent builds a new feature and wires the integration > QA Agent runs tests to prove to itself it works > PR merges, feature goes live, product updates itself in real time It's like the nontechnical stakeholder wrote the code without even knowing it. Coding agents are great at turning good development plans into code, and they're getting better at turning context into good development plans in collaboration with professional engineers. But nontechnical people are capped on what they can build without product people and engineers. The bridge that takes nontechnical people from vibe coding basic apps to building production AI tools that run on first party context is FDEs. Our new FDE Factory gives you the system to go from idea to production. Context Creation is the first and most important step in our FDE lifecycle, and we just automated it. Now clients get the right agents and tools built for them, customized to their unique business and encoded with their expertise. PS: If you're building AI agents within your company, reply "Playbook" and I'll DM you the entire FDE playbook we've run with 30+ companies. It covers finding high-impact AI use cases, building them, and deploying them across the org.

Mike Fishbein

10,171 Aufrufe • vor 3 Monaten

Google announces PALP Prompt Aligned Personalization of Text-to-Image Models paper page: Content creators often aim to create personalized images using personal subjects that go beyond the capabilities of conventional text-to-image models. Additionally, they may want the resulting image to encompass a specific location, style, ambiance, and more. Existing personalization methods may compromise personalization ability or the alignment to complex textual prompts. This trade-off can impede the fulfillment of user prompts and subject fidelity. We propose a new approach focusing on personalization methods for a single prompt to address this issue. We term our approach prompt-aligned personalization. While this may seem restrictive, our method excels in improving text alignment, enabling the creation of images with complex and intricate prompts, which may pose a challenge for current techniques. In particular, our method keeps the personalized model aligned with a target prompt using an additional score distillation sampling term. We demonstrate the versatility of our method in multi- and single-shot settings and further show that it can compose multiple subjects or use inspiration from reference images, such as artworks. We compare our approach quantitatively and qualitatively with existing baselines and state-of-the-art techniques.

Google announces PALP Prompt Aligned Personalization of Text-to-Image Models paper page: Content creators often aim to create personalized images using personal subjects that go beyond the capabilities of conventional text-to-image models. Additionally, they may want the resulting image to encompass a specific location, style, ambiance, and more. Existing personalization methods may compromise personalization ability or the alignment to complex textual prompts. This trade-off can impede the fulfillment of user prompts and subject fidelity. We propose a new approach focusing on personalization methods for a single prompt to address this issue. We term our approach prompt-aligned personalization. While this may seem restrictive, our method excels in improving text alignment, enabling the creation of images with complex and intricate prompts, which may pose a challenge for current techniques. In particular, our method keeps the personalized model aligned with a target prompt using an additional score distillation sampling term. We demonstrate the versatility of our method in multi- and single-shot settings and further show that it can compose multiple subjects or use inspiration from reference images, such as artworks. We compare our approach quantitatively and qualitatively with existing baselines and state-of-the-art techniques.

AK

90,212 Aufrufe • vor 2 Jahren

This is perhaps the best Raksha Bandhan ad I have seen this year, though, to be honest, it wouldn't qualify as an ad at all - more of a short film. It is adequately predictable, feels like something plucked out of a longer movie, but still managed to make me sit through it because it did the basics really well. For starters, unlike most raksha bandhan ads, the brother doesn't simply bring a solitary rakhi and get it tied. He actually brings the paraphernalia, including the diya... lights the diya, and the rakhi becomes a part of the larger effort, and not the only one (yes, it is the most visible part). The act reminded me of this 2025 Diwali ad, by the way: And his response to 'money' is where I thought the ad's writer did give it some thought. It was a charming response, fully in context of the situation, with a good sense of humor. And the camera panning to the constable was a nice touch too! Why a short film, and not an ad? Because the brand connect is literally zero. Swap any brand in the ad and it would equally... not fit :) The agency doesn't make any attempt to integrate anything about the product inside the narrative. To be fair, they cannot - it's that kind of a product 🤷‍♂️ At least if it was a brand of washing detergent, or cooking oil, or plywood... or something, they may have considered adding something about the product inside the narrative... and still, it would look weak/forced. It feels like the story was conceived as a standalone idea removed from any advertising context, and just shot because someone could bankroll this from the/a brand. #advertising #marketing #creativity

Karthik 🇮🇳

34,629 Aufrufe • vor 29 Tagen