Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

📢MeshPad: Interactive Sketch-Conditioned Artist-Designed Mesh Generation and Editing📢 Users can interactively design 3D models just from a sketch-based interface - check out the demo :) We break down the design process into addition with an autoregressive generator and deletion operations enabled by a classifier. To speed-up predictions, we propose... show more

Matthias Niessner

48,516 subscribers

30,032 Aufrufe • vor 1 Jahr •via X (Twitter)

Bildung Kunst Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,849 Aufrufe • vor 1 Jahr

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

MrNeRF

17,062 Aufrufe • vor 1 Jahr

GPT-5.6 Sol is unbelievably good at creating and editing videos. It can do motion design, product demos, and animations like this one I made by simply giving it a screen recording. GPT 5.6 has the best design taste and significantly outperforms Fable, which relies heavily on repetitive design patterns. To help you experiment with video editing on it, we just launched a collection of 100 ready-to-use skills that show what’s possible and help you get started with video editing using GPT-5.6. These skills can create anything from motion graphics launch videos for your product to a 3B1B-style science explainer video. You can also use them to edit existing videos: add captions, generate motion graphics, create voiceovers, redesign visual styles, translate into new languages, and much more. If you want access to the full library, comment “VIDEO SKILLS” and I’ll share it with you. (You'll have to follow me so I can DM you.)

GPT-5.6 Sol is unbelievably good at creating and editing videos. It can do motion design, product demos, and animations like this one I made by simply giving it a screen recording. GPT 5.6 has the best design taste and significantly outperforms Fable, which relies heavily on repetitive design patterns. To help you experiment with video editing on it, we just launched a collection of 100 ready-to-use skills that show what’s possible and help you get started with video editing using GPT-5.6. These skills can create anything from motion graphics launch videos for your product to a 3B1B-style science explainer video. You can also use them to edit existing videos: add captions, generate motion graphics, create voiceovers, redesign visual styles, translate into new languages, and much more. If you want access to the full library, comment “VIDEO SKILLS” and I’ll share it with you. (You'll have to follow me so I can DM you.)

Akash Anand

513,644 Aufrufe • vor 1 Monat

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢 We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive than classical 3DMMs such as FLAME. Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control. Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity. Key to successful and generalized training of our ViT-based network are: (1) large-scale registration of existing 3D head datasets, and (2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals. Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization. 🌍 🎥 Great work by Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢 We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive than classical 3DMMs such as FLAME. Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control. Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity. Key to successful and generalized training of our ViT-based network are: (1) large-scale registration of existing 3D head datasets, and (2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals. Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization. 🌍 🎥 Great work by Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen

Matthias Niessner

37,850 Aufrufe • vor 7 Monaten

At Uber, design source of truth was a big problem for design and engineering teams. No one knew what the app truly looked like. So every week, a group of designers would sit in a meeting room and check that engineers had correctly implemented the Figma designs. Now coding agents have increased the throughput of changes by an order of magnitude. Keeping up manually has become impossible. Here we used the Revyl to map every state in Ubert (demo uber), navigating our mobile use agent on a cloud iOS simulator. This would have taken a team of designers tens of hours to recreate by hand. We did it in under an hour, asynchronously, with a simple prompt. Let your team see what your users actually see. Empower coding agents to ship a delightful experience with no blindspots. Get started with our free trial and create a map for your own app.

At Uber, design source of truth was a big problem for design and engineering teams. No one knew what the app truly looked like. So every week, a group of designers would sit in a meeting room and check that engineers had correctly implemented the Figma designs. Now coding agents have increased the throughput of changes by an order of magnitude. Keeping up manually has become impossible. Here we used the Revyl to map every state in Ubert (demo uber), navigating our mobile use agent on a cloud iOS simulator. This would have taken a team of designers tens of hours to recreate by hand. We did it in under an hour, asynchronously, with a simple prompt. Let your team see what your users actually see. Empower coding agents to ship a delightful experience with no blindspots. Get started with our free trial and create a map for your own app.

Anam Hira

203,496 Aufrufe • vor 1 Monat

Excited that our paper StreamdiffusionV2 received the Best Research Paper Award at #MLSys26! 🚀Video generation is quickly moving from demos to production-facing workloads. It is no longer a turn-based pipeline but should be a streaming pipeline to interact with users. 📖Our project page: and paper: 👂Come join the talk if you are interested in streaming video generation. Our talk will be at the Research Track Oral Presentation: Best Paper Session on Tue 8:45AM at #MLSys26 , I will talk about how we attacked the efficiency and quality challenges. Hope to see you there! ❤️Huge thanks to all authors! This work would not have been possible without the incredible effort from the entire team. Big shout out to Tianrui Feng, Zhi Li, Shuo Yang , Haocheng Xi, Muyang Li , lllyasviel , Xiuyu Li , Keting Yang, Kelly Peng, Song Han , Maneesh Agrawala, Kurt Keutzer , and あき先生(Aki)

Excited that our paper StreamdiffusionV2 received the Best Research Paper Award at #MLSys26! 🚀Video generation is quickly moving from demos to production-facing workloads. It is no longer a turn-based pipeline but should be a streaming pipeline to interact with users. 📖Our project page: and paper: 👂Come join the talk if you are interested in streaming video generation. Our talk will be at the Research Track Oral Presentation: Best Paper Session on Tue 8:45AM at #MLSys26 , I will talk about how we attacked the efficiency and quality challenges. Hope to see you there! ❤️Huge thanks to all authors! This work would not have been possible without the incredible effort from the entire team. Big shout out to Tianrui Feng, Zhi Li, Shuo Yang , Haocheng Xi, Muyang Li , lllyasviel , Xiuyu Li , Keting Yang, Kelly Peng, Song Han , Maneesh Agrawala, Kurt Keutzer , and あき先生(Aki)

Chenfeng_X

59,315 Aufrufe • vor 3 Monaten

Introducing Kaleido💮 from AI at Meta — a universal generative neural rendering engine for photorealistic, unified object and scene view synthesis. Kaleido is built on a simple but powerful design philosophy: 3D perception is a form of visual common sense. Following this idea, we formulate rendering purely as a sequence-to-sequence generation problem, successfully unifying neural rendering with the architecture principles behind modern language and video models. Unlike traditional neural rendering methods, Kaleido learns 3D purely in a data-driven way, without explicit 3D representations or structures. It acquires spatial understanding directly through large-scale video pretraining, then multi-view 3D data finetuning, inspired by how LLMs acquire textual common sense from large corpora before specialising in domains like coding. Through extensive ablations, we progressively modernised the architecture design and training strategies and tackled key scaling challenges in sequence-to-sequence generative rendering, arriving at a design that’s simple, versatile, and scalable. Kaleido significantly outperforms prior generative models in few-view settings, and remarkably is the first zero-shot generative method matches InstantNGP-level rendering quality in multi-view settings. We view Kaleido also as an alternative step towards world modeling that flexibly spans a spectrum of “realities": with many views, it faithfully reconstructs grounded reality; with fewer views, it imagines plausible unseen details. 🔗 Explore more results and paper:

Introducing Kaleido💮 from AI at Meta — a universal generative neural rendering engine for photorealistic, unified object and scene view synthesis. Kaleido is built on a simple but powerful design philosophy: 3D perception is a form of visual common sense. Following this idea, we formulate rendering purely as a sequence-to-sequence generation problem, successfully unifying neural rendering with the architecture principles behind modern language and video models. Unlike traditional neural rendering methods, Kaleido learns 3D purely in a data-driven way, without explicit 3D representations or structures. It acquires spatial understanding directly through large-scale video pretraining, then multi-view 3D data finetuning, inspired by how LLMs acquire textual common sense from large corpora before specialising in domains like coding. Through extensive ablations, we progressively modernised the architecture design and training strategies and tackled key scaling challenges in sequence-to-sequence generative rendering, arriving at a design that’s simple, versatile, and scalable. Kaleido significantly outperforms prior generative models in few-view settings, and remarkably is the first zero-shot generative method matches InstantNGP-level rendering quality in multi-view settings. We view Kaleido also as an alternative step towards world modeling that flexibly spans a spectrum of “realities": with many views, it faithfully reconstructs grounded reality; with fewer views, it imagines plausible unseen details. 🔗 Explore more results and paper:

Shikun Liu

22,389 Aufrufe • vor 10 Monaten

AI Is Moving Beyond “Generating Videos” — Toward “Generating Worlds” Over the past two years, AI video models have advanced at an astonishing pace. From Runway and Pika to Sora and Veo, AI-generated videos have become increasingly realistic and more consistent with the physical laws of the real world. Many people believe the next objective is simply to generate videos that are longer, sharper, and more lifelike. But if we take a step back, we can see that the real transformation is not happening in video itself. It is happening in world models. What Is a World Model? In 1943, psychologist Kenneth Craik proposed an idea that would influence artificial intelligence research for decades. He argued that the human brain does not merely react to the outside world. Instead, it maintains an internal model of how the world works. Because we have this internal model, we can predict the outcome of an action before we actually take it. Before crossing a road, we estimate whether a car will pass by. Before catching a ball, we predict its trajectory. These abilities come from continuously simulating the world in our minds, rather than relying entirely on trial and error. This idea later became known by a more formal term: World Model. A world model does not describe a single image or a fixed video clip. It is an internal representation capable of continuously simulating the rules and dynamics of the real world. Why Is AI Research Turning Toward World Models? Because predicting “what comes next” is becoming increasingly central to how AI systems work. Language models predict the next token. Image models predict the next step in the denoising process. Video models predict the next frame. A world model, however, attempts to predict something broader: What should the world look like in the next moment? In 2018, David Ha and Jürgen Schmidhuber proposed in their paper World Models that an intelligent agent could first learn a model of the world, and then use that internal model to plan its actions. The Dreamer series later demonstrated that many complex tasks could be learned by training agents inside an “imagined world.” At the same time, the development of video models such as Sora and Veo led researchers to another realization: A model capable of continuously generating video has already learned, at least implicitly, many of the rules governing the real world. As a result, these two research directions have gradually begun to converge. But Video Is Not Yet a World This is where the distinction is often misunderstood. For a world model to support meaningful real-time interaction, it must solve several critical problems. Most video models today are essentially answering one question: What should the next frame look like? A true world model needs to answer much more: What happens if I take one step forward? If I walk behind a building and then return, will the building still be there? If I suddenly change the camera angle, will the entire space remain consistent? If I enter a command such as: “Summon a dragon.” Will the world respond immediately? In other words, a world model must do more than generate content. It must understand space. It must understand time. It must understand causality. And it must understand interaction. Moving from watching to participating is where the real difficulty of world models begins. World Models Are Entering the Interactive Era One of the latest attempts in this direction is Alaya World, recently open-sourced by Alaya World, or Alaya Lab. Instead of generating a fixed video clip, it generates a world that users can explore in real time. Users can begin with text, an image, or a video, enter the generated scene, move freely through it, and introduce new prompts at any moment during generation. The world responds immediately. According to the publicly released information, Alaya World provides: Real-time streaming generation at 720p and 24 FPS Stable continuous exploration for more than one minute The ability to switch prompts and trigger skills or events during generation Model weights and inference code released under the Apache 2.0 License Training code and datasets planned for future release What makes these capabilities important is not simply the technical specifications. It is that the generated “world” can now support continuous interaction. The official demo shows that users can genuinely control, transform, and explore the generated environment. AI Is Evolving From a Tool Into an Environment Over the past few years, most discussions around AI have focused on content generation. Generating text. Generating images. Generating videos. But world models raise a fundamentally different question: Can AI generate an environment that people can inhabit, explore, and continuously evolve? If the answer is yes, the impact will extend far beyond video generation. Game development, robotics training, embodied intelligence, digital twins, virtual production, and many other fields could be transformed by the development of world models. World models are still at a very early stage. Yet from Craik’s proposal of an internal mental model more than eighty years ago to the emergence of today’s interactive world-generation systems, a clear evolutionary path is beginning to take shape. Perhaps what AI is ultimately learning has never been limited to images, videos, or language. Perhaps it is learning the world itself. References GitHub: Technical Report:

AI Is Moving Beyond “Generating Videos” — Toward “Generating Worlds” Over the past two years, AI video models have advanced at an astonishing pace. From Runway and Pika to Sora and Veo, AI-generated videos have become increasingly realistic and more consistent with the physical laws of the real world. Many people believe the next objective is simply to generate videos that are longer, sharper, and more lifelike. But if we take a step back, we can see that the real transformation is not happening in video itself. It is happening in world models. What Is a World Model? In 1943, psychologist Kenneth Craik proposed an idea that would influence artificial intelligence research for decades. He argued that the human brain does not merely react to the outside world. Instead, it maintains an internal model of how the world works. Because we have this internal model, we can predict the outcome of an action before we actually take it. Before crossing a road, we estimate whether a car will pass by. Before catching a ball, we predict its trajectory. These abilities come from continuously simulating the world in our minds, rather than relying entirely on trial and error. This idea later became known by a more formal term: World Model. A world model does not describe a single image or a fixed video clip. It is an internal representation capable of continuously simulating the rules and dynamics of the real world. Why Is AI Research Turning Toward World Models? Because predicting “what comes next” is becoming increasingly central to how AI systems work. Language models predict the next token. Image models predict the next step in the denoising process. Video models predict the next frame. A world model, however, attempts to predict something broader: What should the world look like in the next moment? In 2018, David Ha and Jürgen Schmidhuber proposed in their paper World Models that an intelligent agent could first learn a model of the world, and then use that internal model to plan its actions. The Dreamer series later demonstrated that many complex tasks could be learned by training agents inside an “imagined world.” At the same time, the development of video models such as Sora and Veo led researchers to another realization: A model capable of continuously generating video has already learned, at least implicitly, many of the rules governing the real world. As a result, these two research directions have gradually begun to converge. But Video Is Not Yet a World This is where the distinction is often misunderstood. For a world model to support meaningful real-time interaction, it must solve several critical problems. Most video models today are essentially answering one question: What should the next frame look like? A true world model needs to answer much more: What happens if I take one step forward? If I walk behind a building and then return, will the building still be there? If I suddenly change the camera angle, will the entire space remain consistent? If I enter a command such as: “Summon a dragon.” Will the world respond immediately? In other words, a world model must do more than generate content. It must understand space. It must understand time. It must understand causality. And it must understand interaction. Moving from watching to participating is where the real difficulty of world models begins. World Models Are Entering the Interactive Era One of the latest attempts in this direction is Alaya World, recently open-sourced by Alaya World, or Alaya Lab. Instead of generating a fixed video clip, it generates a world that users can explore in real time. Users can begin with text, an image, or a video, enter the generated scene, move freely through it, and introduce new prompts at any moment during generation. The world responds immediately. According to the publicly released information, Alaya World provides: Real-time streaming generation at 720p and 24 FPS Stable continuous exploration for more than one minute The ability to switch prompts and trigger skills or events during generation Model weights and inference code released under the Apache 2.0 License Training code and datasets planned for future release What makes these capabilities important is not simply the technical specifications. It is that the generated “world” can now support continuous interaction. The official demo shows that users can genuinely control, transform, and explore the generated environment. AI Is Evolving From a Tool Into an Environment Over the past few years, most discussions around AI have focused on content generation. Generating text. Generating images. Generating videos. But world models raise a fundamentally different question: Can AI generate an environment that people can inhabit, explore, and continuously evolve? If the answer is yes, the impact will extend far beyond video generation. Game development, robotics training, embodied intelligence, digital twins, virtual production, and many other fields could be transformed by the development of world models. World models are still at a very early stage. Yet from Craik’s proposal of an internal mental model more than eighty years ago to the emergence of today’s interactive world-generation systems, a clear evolutionary path is beginning to take shape. Perhaps what AI is ultimately learning has never been limited to images, videos, or language. Perhaps it is learning the world itself. References GitHub: Technical Report:

雪踏乌云

113,347 Aufrufe • vor 1 Monat

How I use Claude Code and Remotion to make animated diagrams. Sorry, it's not a single prompt. 1. Find an input language the model knows well. For example, Mermaid for flowcharts. Claude writes it fluently, so it's my entry point. 2. Use Claude to build components that take that input and bake in the guardrails: design system, animation patterns, layout rules. 3. Now I can describe what I want in plain English e.g. "create a flowchart for the tier check section in the script", and Claude translates it to our input language: ``` flowchart TD t1[Tier 1 read-only] t2[Tier 2 in-project writes] t3[Tier 3 everything else] action[Action] --> t1 action --> t2 action --> t3 t1 --> skip([Skips classifier]) t2 --> skip t3 --> classifier{Classifier} classifier --> approve[Approve] classifier --> deny[Deny] ``` The component handles the rest: layout, styling, node and edge reveals. It also takes events for follow-ups like the trace dot that follows a path and lights up nodes. 4. To finish it off, I wrapped the board in a separate CRT shader component. It really helps to have a shared vocabulary with your agent. When I say "rise in fast on enter", it knows I mean fade in while translating up, from a set offset, faster than the default duration, with a specific bezier curve. For common language inspo: look into Matt Pocock `/grill-with-docs` and by Emil Kowalski and Glenn Hitchcock.

How I use Claude Code and Remotion to make animated diagrams. Sorry, it's not a single prompt. 1. Find an input language the model knows well. For example, Mermaid for flowcharts. Claude writes it fluently, so it's my entry point. 2. Use Claude to build components that take that input and bake in the guardrails: design system, animation patterns, layout rules. 3. Now I can describe what I want in plain English e.g. "create a flowchart for the tier check section in the script", and Claude translates it to our input language: ``` flowchart TD t1[Tier 1 read-only] t2[Tier 2 in-project writes] t3[Tier 3 everything else] action[Action] --> t1 action --> t2 action --> t3 t1 --> skip([Skips classifier]) t2 --> skip t3 --> classifier{Classifier} classifier --> approve[Approve] classifier --> deny[Deny] ``` The component handles the rest: layout, styling, node and edge reveals. It also takes events for follow-ups like the trace dot that follows a path and lights up nodes. 4. To finish it off, I wrapped the board in a separate CRT shader component. It really helps to have a shared vocabulary with your agent. When I say "rise in fast on enter", it knows I mean fade in while translating up, from a set offset, faster than the default duration, with a specific bezier curve. For common language inspo: look into Matt Pocock `/grill-with-docs` and by Emil Kowalski and Glenn Hitchcock.

Delba

61,247 Aufrufe • vor 2 Monaten

Claude Fable 5 + Claude Design is f*cking insane 🤯 Anthropic just dropped its most intelligent model ever, and the first thing I pointed it at was email design. I built a complete email campaign design in Claude Design, and the difference is night and day: tighter layouts, cleaner hierarchy, on-brand from the first generation. All inside Claude Design with Fable 5. Perfect for DTC brands and agencies who are still paying email agencies $3-5K/month for campaign designs that take 2 weeks to ship. If your campaign calendar is packed but every new email means briefing a designer, waiting on mockups, sending notes, and waiting again... This workflow eliminates the entire bottleneck: → Load your brand design system into Claude Design once (colors, fonts, logo, button styling) → Switch the model to Claude Fable 5 — Anthropic's new state-of-the-art model with the best vision of any AI → Prompt the campaign email section by section: header, hero, headline, offer block, CTA → Fable 5 nails layout and brand details that older models fumbled → Iterate inline — swap images, adjust styling, color-pick directly in the canvas → Export the finished email and hand off to your ESP No briefing a designer. No 2-week turnaround on a single campaign. No paying an agency $4K/month for 4 emails. What you get: → Campaign emails designed in minutes, not weeks → A reusable design system every new email pulls from automatically → Noticeably smarter design decisions from Fable 5's upgraded vision → Full inline editing before anything touches your ESP Built 100% with Claude Design + Claude Fable 5. I recorded a full walkthrough showing exactly how this works. Want it for free? > Like this post > Comment "FABLE" And I'll send it over (must be following so I can DM)

Mike Futia

42,932 Aufrufe • vor 2 Monaten

Systems created from racism cannot be reformed. That is why we keep seeing the same thing happen over and over again… with police and with ICE. An unarmed Black man, in Milwaukee, was shot in the head three times after being tased and restrained by five officers. Meanwhile, just 80 miles away, a white man was actively on top of a police officer, assaulting him… and was taken into custody without being shot. We have seen this same disparity play out over and over again. And we will continue to see this happen because… American policing is deeply rooted in systems created to control and hunt down Black populations... and ICE was created from a system rooted in racial exclusion, and the criminalization of immigrants. And we see the consequences of that history every single day. The problem is not just a few bad officers/agents, or a lack of training. The problem is not just that the wrong person was hired… These systems were built on racism. They were built to use state violence against certain people. You cannot reform a system whose foundation is rotten. You cannot train racism out of a system created by racism. You cannot reform your way out of a structure that is operating how it was designed to operate. At some point, we have to stop asking how to make these systems work better… And we have to start asking why we keep defending systems that were never designed to protect people in the first place.

Systems created from racism cannot be reformed. That is why we keep seeing the same thing happen over and over again… with police and with ICE. An unarmed Black man, in Milwaukee, was shot in the head three times after being tased and restrained by five officers. Meanwhile, just 80 miles away, a white man was actively on top of a police officer, assaulting him… and was taken into custody without being shot. We have seen this same disparity play out over and over again. And we will continue to see this happen because… American policing is deeply rooted in systems created to control and hunt down Black populations... and ICE was created from a system rooted in racial exclusion, and the criminalization of immigrants. And we see the consequences of that history every single day. The problem is not just a few bad officers/agents, or a lack of training. The problem is not just that the wrong person was hired… These systems were built on racism. They were built to use state violence against certain people. You cannot reform a system whose foundation is rotten. You cannot train racism out of a system created by racism. You cannot reform your way out of a structure that is operating how it was designed to operate. At some point, we have to stop asking how to make these systems work better… And we have to start asking why we keep defending systems that were never designed to protect people in the first place.

Jesus Freakin Congress

48,590 Aufrufe • vor 24 Tagen

When creating this cut for SCP:GALLIONIC's trailer we started out with a 3D layout, where the general movement and camera placements are decided. Then animation and Backgrounds are worked on in tandem. Creating the background was fairly straightforward (just define the structure, textures and lighting) [done by Brando Leon Rustici - (So So Animations) ] Animation however, was definitely a bit trickier, but Jag アニメーター really knocked it out of the park: especially with such complicated designs and number of individual objects, it's always good (as he's done) to break it down first to just "blobs", to get the movement down and after that actually drawing the characters properly. Mikey Nowack, other than being being responsible for SCP-1762's design, also took care of the colour script for this cut, and together with Wing (in charge of compositing) brought the final scene to life! Go check out the full trailer if you have not already! #scp #indie_anime

When creating this cut for SCP:GALLIONIC's trailer we started out with a 3D layout, where the general movement and camera placements are decided. Then animation and Backgrounds are worked on in tandem. Creating the background was fairly straightforward (just define the structure, textures and lighting) [done by Brando Leon Rustici - (So So Animations) ] Animation however, was definitely a bit trickier, but Jag アニメーター really knocked it out of the park: especially with such complicated designs and number of individual objects, it's always good (as he's done) to break it down first to just "blobs", to get the movement down and after that actually drawing the characters properly. Mikey Nowack, other than being being responsible for SCP-1762's design, also took care of the colour script for this cut, and together with Wing (in charge of compositing) brought the final scene to life! Go check out the full trailer if you have not already! #scp #indie_anime

soem studios

22,673 Aufrufe • vor 5 Monaten

[Most robots react. This one thinks a step ahead.] Ant Group's Robbyant just published LingBot-VA 2.0 — a video-action foundation model built from scratch for robot control, not fine-tuned from a video generator. The usual approach takes a video generator made for content creation and bolts a robot policy onto it. LingBot-VA 2.0 argues that's the wrong starting point, and pretrains the whole causal stack natively instead. What stands out: → Foresight Reasoning — the robot predicts the next action chunk while executing the current one, then overwrites the imagined frame with the real observation. Prediction and execution stop waiting on each other. → 927 ms → 142 ms per chunk, across four cumulative optimizations. That lifts asynchronous control from 35 Hz to 225 Hz — a 6.5× speedup. → One shared latent space. A semantic visual-action tokenizer puts world states and actions in the same coordinates, so unlabeled web video carries action-relevant signal. → Sparse MoE video stream — 128 experts, top-8 routing. Roughly 2.5B of ~15.3B parameters fire per token. → Few-shot by design — adapts from 10–15 demonstrations, and a human demo video can replace the text instruction entirely. Full breakdown: Paper: Project Page: Robbyant Ant Group

[Most robots react. This one thinks a step ahead.] Ant Group's Robbyant just published LingBot-VA 2.0 — a video-action foundation model built from scratch for robot control, not fine-tuned from a video generator. The usual approach takes a video generator made for content creation and bolts a robot policy onto it. LingBot-VA 2.0 argues that's the wrong starting point, and pretrains the whole causal stack natively instead. What stands out: → Foresight Reasoning — the robot predicts the next action chunk while executing the current one, then overwrites the imagined frame with the real observation. Prediction and execution stop waiting on each other. → 927 ms → 142 ms per chunk, across four cumulative optimizations. That lifts asynchronous control from 35 Hz to 225 Hz — a 6.5× speedup. → One shared latent space. A semantic visual-action tokenizer puts world states and actions in the same coordinates, so unlabeled web video carries action-relevant signal. → Sparse MoE video stream — 128 experts, top-8 routing. Roughly 2.5B of ~15.3B parameters fire per token. → Few-shot by design — adapts from 10–15 demonstrations, and a human demo video can replace the text instruction entirely. Full breakdown: Paper: Project Page: Robbyant Ant Group

Marktechpost AI

196,499 Aufrufe • vor 1 Monat

What does it look like when a travel brand starts from mythology and ends up inside a booking conversation? Francis Davidson, co-founder of Sonder, had a specific belief: that the trip you imagine and the trip you take shouldn't be two different things. Odessia was created to close that distance and excited for Little Plains to have been along for this journey. Odessia is named after a fictional Greek goddess of travel, one who covers all of it: the desire to go, the dawning clarity of what you actually want, the pleasure of the trip itself, and the memories that outlast it. She's been everywhere, which means she has taste without pretension. She listens as carefully as she advises. The identity flows directly from her. We drew the wordmark from engraved Greek letterforms, balancing geometric precision with a humanist touch. Odessia's face is laurel-crowned and sits at the center of the mark. We then built a coin system inspired by ancient Greek currency, animated and sculptural, designed to spin and catch light every time a traveler hits a milestone. We countered lots of warm antiquity, clay-based colors, with a deep ocean-blue, and warm sunrise gold. We wanted to give off the energy of: Dream wild. Plan easy. From this foundation, we moved into helping Francis (a design/dev unicorn!) with the product. We worked collaboratively to create conversational interfaces that lead with destinations, not technology. One conversation handles flights, hotels, itineraries, loyalty points, and booking from start to finish. The Odessia Collection adds 2,000+ luxury properties with VIP treatment built in: upgrades, credits, early check-in, for travelers who don't have a personal assistant but travel like they do. Travelers trust AI for the outcome, not the process. So Odessia built a product that shows the wonder and makes the optimization invisible. ~ Little Plains helped Francis, the amazing Emma Bates and the Odessia team with naming, brand strategy, identity, brand voice, product UX and UI, conversational interface design, icon and coin system, marketing site, and launch assets. Grateful to the Odessia team for the trust. Bon Voyage!

Emmett

326,456 Aufrufe • vor 2 Monaten

Everyone's sleeping on image-to-3D AI models. They can make your app look incredibly unique, with just a little effort. Here's how. This is my calorie tracker, built in a week with nothing but prompting. Just Claude Code + a couple APIs. The visuals are all AI-generated. I'll be sharing the full workflow + all the crazy technical stuff Claude and I did to make this work, so nobody has to struggle through it like me. Deep dive coming soon! Till then, this is the high-level idea: 1. Get a clean image of the food (or whatever your asset is) - In my app, the user describes foods via text, or attaches images (or both) - If text, an LLM extracts the food description and formats it into a specific prompt I tuned for this design, and we generate an image using Z-Image Turbo through fal - If image, we do the same thing but with FLUX.2 [dev] to edit the user image into our reference design - Originally, both used Google Nano Banana, but switching to open models cut costs and latency a ton 2. Gaussian splatting (2D image → 3D model) - I tried various 2D-to-3D options on fal and ended up with TripoSplat as my preferred balance of speed, cost, latency; this turns an image into a 3D model that looks super high quality (link below) - The app displays the 2D image while our backend generates the 3D splat - We "groom" the splat to reduce size and load time by culling low-opacity/scale points 3. Render efficiently on device Originally, it looked great but ran at 10 FPS. Getting to 120 FPS was a crazy journey. TL;DR: - SwiftUI had to go; it forced us to render each asset in independent MTKViews, which wasn't workable - Instead, we composite every dish into one full-bleed CAMetalLayer using MetalSplatter (link below) - We had to make some optimizations within MetalSplatter's code too, to reduce the overhead of sorting points per render Then I added some finishing touches like the subtle rotation and parallax as they move around. I think it turned out pretty cool :) Overall, this took some effort, but we still got it done in less than a day. Hopefully your agent can follow in the footsteps of mine and do it much faster. Keep an eye out for the bigger writeup, which'll give your agent everything it needs. If you have any questions, drop em below!

Everyone's sleeping on image-to-3D AI models. They can make your app look incredibly unique, with just a little effort. Here's how. This is my calorie tracker, built in a week with nothing but prompting. Just Claude Code + a couple APIs. The visuals are all AI-generated. I'll be sharing the full workflow + all the crazy technical stuff Claude and I did to make this work, so nobody has to struggle through it like me. Deep dive coming soon! Till then, this is the high-level idea: 1. Get a clean image of the food (or whatever your asset is) - In my app, the user describes foods via text, or attaches images (or both) - If text, an LLM extracts the food description and formats it into a specific prompt I tuned for this design, and we generate an image using Z-Image Turbo through fal - If image, we do the same thing but with FLUX.2 [dev] to edit the user image into our reference design - Originally, both used Google Nano Banana, but switching to open models cut costs and latency a ton 2. Gaussian splatting (2D image → 3D model) - I tried various 2D-to-3D options on fal and ended up with TripoSplat as my preferred balance of speed, cost, latency; this turns an image into a 3D model that looks super high quality (link below) - The app displays the 2D image while our backend generates the 3D splat - We "groom" the splat to reduce size and load time by culling low-opacity/scale points 3. Render efficiently on device Originally, it looked great but ran at 10 FPS. Getting to 120 FPS was a crazy journey. TL;DR: - SwiftUI had to go; it forced us to render each asset in independent MTKViews, which wasn't workable - Instead, we composite every dish into one full-bleed CAMetalLayer using MetalSplatter (link below) - We had to make some optimizations within MetalSplatter's code too, to reduce the overhead of sorting points per render Then I added some finishing touches like the subtle rotation and parallax as they move around. I think it turned out pretty cool :) Overall, this took some effort, but we still got it done in less than a day. Hopefully your agent can follow in the footsteps of mine and do it much faster. Keep an eye out for the bigger writeup, which'll give your agent everything it needs. If you have any questions, drop em below!

Anshu

19,931 Aufrufe • vor 1 Monat

Dear Tarun Chitra 1. We are the original creators of DeSci back in 2016. What DeSci has become today is largely unrelated with its original model of producing rigorous peer-reviewed scientific studies published in reputable medical journals. The model we introduced. We are tirelessly fighting against pseudoscience, and we are showing the world that people can understand the difference between legit science and pseudoscience with the success of $INNBCV. Yes, meritocracy is possible in crypto. Even against all odds. 2. We are the only project in the entire crypto space that ever funded, performed, and published highly innovative HIV cure research ( We are the project that produced the first peer-reviewed study on blockchain-based biomedical data storage in the world’s most reputable scientific network, Springer Nature ( $INNBCV is not for the privileged few; it is for the many. We resisted all the pressure from those who wanted us to provide big allocations to VIPs of other DAOs “because it is good for the marketing” and put our users first, ensuring a fair launch, a launch for the people, and they turned $70k into $2,000,000. $INNBCV shows that you can have a sustainable model, provided you are backed by actual science. And thanks to the amazing guys at daos.fun baoskee and Solana community. Behind our project there is the sweat and blood of years of work to produce publications in the most reputable medical journals. Just to put things into perspective, it took us 3 years to publish our latest work in Springer Nature. 3. Unlike many other projects, we had no ICO/VCs, meaning we had to prove ourselves every single day because we are only supported by our community. If we deliver products, we survive; it is either publish or perish for us, and that’s why we have such a close connection to our community. $INNBCV is a struggler, $INNBCV is a survivor, $INNBCV is not for the privilege of the few but for the people. Our community makes it possible by supporting us. You guys are the real heroes.

Dear Tarun Chitra 1. We are the original creators of DeSci back in 2016. What DeSci has become today is largely unrelated with its original model of producing rigorous peer-reviewed scientific studies published in reputable medical journals. The model we introduced. We are tirelessly fighting against pseudoscience, and we are showing the world that people can understand the difference between legit science and pseudoscience with the success of $INNBCV. Yes, meritocracy is possible in crypto. Even against all odds. 2. We are the only project in the entire crypto space that ever funded, performed, and published highly innovative HIV cure research ( We are the project that produced the first peer-reviewed study on blockchain-based biomedical data storage in the world’s most reputable scientific network, Springer Nature ( $INNBCV is not for the privileged few; it is for the many. We resisted all the pressure from those who wanted us to provide big allocations to VIPs of other DAOs “because it is good for the marketing” and put our users first, ensuring a fair launch, a launch for the people, and they turned $70k into $2,000,000. $INNBCV shows that you can have a sustainable model, provided you are backed by actual science. And thanks to the amazing guys at daos.fun baoskee and Solana community. Behind our project there is the sweat and blood of years of work to produce publications in the most reputable medical journals. Just to put things into perspective, it took us 3 years to publish our latest work in Springer Nature. 3. Unlike many other projects, we had no ICO/VCs, meaning we had to prove ourselves every single day because we are only supported by our community. If we deliver products, we survive; it is either publish or perish for us, and that’s why we have such a close connection to our community. $INNBCV is a struggler, $INNBCV is a survivor, $INNBCV is not for the privilege of the few but for the people. Our community makes it possible by supporting us. You guys are the real heroes.

InnovativeBioresearch🇮🇹

10,867 Aufrufe • vor 1 Jahr

A look back at our mission to do /more. 🔥🛠️ Months ago, we entered a new growth phase for xLaunchpad, driven by a simple commitment: to do more. More exciting startups, more platform improvements, and a deeper, more meaningful connection with our community. /new We’ve introduced multiple initiatives to make participation in xLaunchpad more accessible, while still keeping EGLD stakers at the core of the experience: 🔸 Challenges Portal – Built in partnership with ᕈulsar Money, this platform allows users to earn lottery tickets by completing simple tasks. 🔸 Creators Program – A reward system for users who create valuable content about xLaunchpad and its projects. 🔸 xLaunchpad Users – Loyal participants from previous launches now have a chance to earn lottery tickets. 🔸 Project Users – Projects launching on xLaunchpad can distribute lottery tickets to eligible users based on set criteria. 🔸 EGLD Holders – Tickets can now be earned simply by holding EGLD in a wallet. Beyond these initiatives, we’ve also strengthened relationships with key community members, empowering them to write educational content about upcoming startups with full creative control on their side. And of course, we’ve introduced more projects and focused on accelerating launch timelines. /better The platform and lottery system have undergone continuous improvements, such as making participation more accessible as detailed above: 🔸 Faster EGLD Reclaims - Users now get their EGLD back more quickly after participation. 🔸 Improved Communication – Blog & X Articles covering both general topics and project-specific updates. Increased community discussions and more structured feedback collection. 🔸 Website Enhancements - Various UI/UX tweaks, with /more to come 👀. 🔸 Integrating AshSwap - Users could buy lottery tickets for the last project with any tokens. 🔸 KYC Improvements - Streamlined processes, with /more to come 👀. Additionally, we’ve successfully co-launched projects with other launchpads and supported startups in securing partnerships, listings, business deals, and legal guidance. xLaunchpad remains one of the most compliant launchpads, ensuring projects align with MiCA regulations and broader industry standards. /next Looking ahead, our main focus remains on the product and the community. Every improvement is guided by real user interactions and feedback, ensuring xLaunchpad continues to evolve. At its core, our mission revolves around two key goals: 1️⃣ Bringing exciting startups to life 2️⃣ Providing users with investment opportunities on the best terms For xLaunchpad and its community to thrive together, it’s essential to recognize these two pillars while understanding the platform’s bigger purpose: making something possible that wasn’t before - allowing retail users to participate in the early launch of promising projects and benefit through multiple avenues. From investments to early adoption, and from unlocking key benefits to shaping young startups. We cannot stress enough how much effort the team puts into achieving these key goals, and we’re grateful to have you by our side on this mission. /more

A look back at our mission to do /more. 🔥🛠️ Months ago, we entered a new growth phase for xLaunchpad, driven by a simple commitment: to do more. More exciting startups, more platform improvements, and a deeper, more meaningful connection with our community. /new We’ve introduced multiple initiatives to make participation in xLaunchpad more accessible, while still keeping EGLD stakers at the core of the experience: 🔸 Challenges Portal – Built in partnership with ᕈulsar Money, this platform allows users to earn lottery tickets by completing simple tasks. 🔸 Creators Program – A reward system for users who create valuable content about xLaunchpad and its projects. 🔸 xLaunchpad Users – Loyal participants from previous launches now have a chance to earn lottery tickets. 🔸 Project Users – Projects launching on xLaunchpad can distribute lottery tickets to eligible users based on set criteria. 🔸 EGLD Holders – Tickets can now be earned simply by holding EGLD in a wallet. Beyond these initiatives, we’ve also strengthened relationships with key community members, empowering them to write educational content about upcoming startups with full creative control on their side. And of course, we’ve introduced more projects and focused on accelerating launch timelines. /better The platform and lottery system have undergone continuous improvements, such as making participation more accessible as detailed above: 🔸 Faster EGLD Reclaims - Users now get their EGLD back more quickly after participation. 🔸 Improved Communication – Blog & X Articles covering both general topics and project-specific updates. Increased community discussions and more structured feedback collection. 🔸 Website Enhancements - Various UI/UX tweaks, with /more to come 👀. 🔸 Integrating AshSwap - Users could buy lottery tickets for the last project with any tokens. 🔸 KYC Improvements - Streamlined processes, with /more to come 👀. Additionally, we’ve successfully co-launched projects with other launchpads and supported startups in securing partnerships, listings, business deals, and legal guidance. xLaunchpad remains one of the most compliant launchpads, ensuring projects align with MiCA regulations and broader industry standards. /next Looking ahead, our main focus remains on the product and the community. Every improvement is guided by real user interactions and feedback, ensuring xLaunchpad continues to evolve. At its core, our mission revolves around two key goals: 1️⃣ Bringing exciting startups to life 2️⃣ Providing users with investment opportunities on the best terms For xLaunchpad and its community to thrive together, it’s essential to recognize these two pillars while understanding the platform’s bigger purpose: making something possible that wasn’t before - allowing retail users to participate in the early launch of promising projects and benefit through multiple avenues. From investments to early adoption, and from unlocking key benefits to shaping young startups. We cannot stress enough how much effort the team puts into achieving these key goals, and we’re grateful to have you by our side on this mission. /more

xLaunchpad

22,919 Aufrufe • vor 1 Jahr

NHS nightmare update: After three days, my wife still hadn't received an ultrasound or an appointment for one, despite suspected deep vein thrombosis (DVT). She runs out of blood-thinner injections in a few days, but we were just told wasn't considered "Urgent" by the Radiology department, and hasn't been referred DVT clinic, despite the risk to her and the babies. We were told on Monday and Tuesday to expect a call from Scans that day, and to call back if we hadn't heard anything. We call back today, and the below is what we got before being passed onto the switchboard. (Excuse my disheveled appearance.) After being passed between departments, we called Maternity Triage again, and a kind English woman answered the phone, listened to everything that happened, and said she would call us back with news within half an hour. Five minutes later, she called back with a scan appointment for tomorrow morning, and immediate follow-up with Maternity. Couldn't have been more helpful. Hopefully we get to the bottom of this tomorrow, and she gets prescribed the treatment she needs to keep her and the girls safe. In the meantime, my Nan has been taken into hospital and kept overnight for complications related to her chronic blood cancer. It's treatable, but not curable, and by all accounts she's being well attended to. A stressful time for the Tomlinsons. Thanks for all your prayers and support.

NHS nightmare update: After three days, my wife still hadn't received an ultrasound or an appointment for one, despite suspected deep vein thrombosis (DVT). She runs out of blood-thinner injections in a few days, but we were just told wasn't considered "Urgent" by the Radiology department, and hasn't been referred DVT clinic, despite the risk to her and the babies. We were told on Monday and Tuesday to expect a call from Scans that day, and to call back if we hadn't heard anything. We call back today, and the below is what we got before being passed onto the switchboard. (Excuse my disheveled appearance.) After being passed between departments, we called Maternity Triage again, and a kind English woman answered the phone, listened to everything that happened, and said she would call us back with news within half an hour. Five minutes later, she called back with a scan appointment for tomorrow morning, and immediate follow-up with Maternity. Couldn't have been more helpful. Hopefully we get to the bottom of this tomorrow, and she gets prescribed the treatment she needs to keep her and the girls safe. In the meantime, my Nan has been taken into hospital and kept overnight for complications related to her chronic blood cancer. It's treatable, but not curable, and by all accounts she's being well attended to. A stressful time for the Tomlinsons. Thanks for all your prayers and support.

Connor Tomlinson

209,949 Aufrufe • vor 12 Tagen

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to *potentially* giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to potentially giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.

nathan (in sf)

40,440 Aufrufe • vor 7 Monaten