Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

This is huge! A UCLA team managed to build an optical generative model that runs on light instead of GPUs. In their demo, a shallow encoder maps noise into phase patterns, which a free-space optical decoder then transforms into images—digits, fashion, butterflies, faces, even Van Gogh–style art—without any computation... show more

机器之心 JIQIZHIXIN

19,058 subscribers

173,558 просмотров • 9 месяцев назад •via X (Twitter)

Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

StyleDrop: Text-to-Image Generation in Any Style introduce StyleDrop, a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. The proposed method is extremely versatile and captures nuances and details of a user-provided style, such as color schemes, shading, design patterns, and local and global effects. It efficiently learns a new style by fine-tuning very few trainable parameters (less than 1% of total model parameters) and improving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user supplies only a single image that specifies the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop implemented on Muse convincingly outperforms other methods, including DreamBooth and textual inversion on Imagen or Stable Diffusion. paper page:

StyleDrop: Text-to-Image Generation in Any Style introduce StyleDrop, a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. The proposed method is extremely versatile and captures nuances and details of a user-provided style, such as color schemes, shading, design patterns, and local and global effects. It efficiently learns a new style by fine-tuning very few trainable parameters (less than 1% of total model parameters) and improving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user supplies only a single image that specifies the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop implemented on Muse convincingly outperforms other methods, including DreamBooth and textual inversion on Imagen or Stable Diffusion. paper page:

AK

56,376 просмотров • 3 лет назад

I made this $700,000,000 music video in an afternoon on part of my weed budget. No, this is not 100% AI. I wrote the song, Through The Tempest, and recorded the samples which were then layered and mixed into the song. I curated the custom styling by running thousands of images of my own art to make a custom model. That model was then iterated dozens of times to fine tune details into new models. Then from the final model I ran dozens of images to animate with the help of AI. All of this to create something I'm going to be projecting later tonight downtown bringing all of this work into the real world. Enjoy.

I made this $700,000,000 music video in an afternoon on part of my weed budget. No, this is not 100% AI. I wrote the song, Through The Tempest, and recorded the samples which were then layered and mixed into the song. I curated the custom styling by running thousands of images of my own art to make a custom model. That model was then iterated dozens of times to fine tune details into new models. Then from the final model I ran dozens of images to animate with the help of AI. All of this to create something I'm going to be projecting later tonight downtown bringing all of this work into the real world. Enjoy.

BLVCKL!GHT

24,034 просмотров • 4 месяцев назад

Jensen Huang is investing in every photonics company he can find and the reason why tells you everything about where AI is headed (Save this). Lip-Bu Tan, the CEO of Intel says, when he looks for investment opportunities, he looks for the bottleneck and right now, the bottleneck is the interconnect, the pipes that move data between chips inside an AI data center. That is why he backed Credo Semiconductor, Astera Labs and Celestial AI on the optical side. Here is the simple version of what the interconnect bottleneck actually means. Think of an AI data center like a city, the GPUs are the buildings where all the work happens but for those buildings to function, you need roads connecting them, fast roads that can carry enormous traffic without congestion. And those roads are now the single biggest constraint on AI performance. As clusters scale to hundreds of thousands of GPUs, traditional copper wiring is hitting its physical limits and that is where this entire sector comes in. Credo Semiconductor (CRDO) is the most direct pure play on this theme, Credo makes high speed cables and optical chips that connect GPUs inside data center racks. Their revenue tripled in fiscal 2026 to $1.3 billion, growing 272% year over year at its peak and four of the world's largest hyperscalers each individually account for more than 10% of Credo's revenue. Astera Labs (ALAB) solves the connection problem between different chip types. Astera makes the PCIe and connectivity chips that manage data flow between GPUs, CPUs, and memory without errors or slowdowns. Their revenue grew 93% year over year to $308 million in Q1 2026 alone. The optical companies are where the longer-term and potentially larger opportunity lives. Copper has physical limits, you can only push electrical signals so far before the signal degrades, the heat spikes and power consumption explodes. The solution is light, fiber optic connections that move data using photons instead of electrons which is faster, cooler and far more energy efficient. Jensen Huang made this clear at Computex 2026 because copper works as long as physically possible but at greater distances and larger scale, optics takes over. Coherent (COHR) is the most established optical company in this space. Coherent makes the lasers, transceivers, and optical components at the foundation of all fiber optic communications. Nvidia signed a multibillion-dollar purchase commitment and invested $2 billion directly into the company and their customer order books are already extending out to 2028. Marvell (MRVL) is the most comprehensive bet across the entire connectivity stack. Marvell makes chips for optical networking, PCIe switching and custom AI silicon. Jensen Huang called Marvell the next trillion dollar company at Computex 2026 and backed it with a $2 billion Nvidia investment. Marvell also acquired Celestial AI, the exact company Lip-Bu Tan backed for $3.25 billion, gaining photonic fabric technology delivering 16 terabits per second of bandwidth. Lumentum (LITE), Corning (GLW), and Ciena (CIEN) round out the major public names. Lumentum received a $2 billion Nvidia investment for laser and photonics components. Corning known mostly for phone glass received $500 million from Nvidia for optical connectivity work and is up over 100% year to date. Ciena runs the optical networking systems between data centers and is seeing analyst price targets raised on the back of the AI optics boom. Every time a hyperscaler spends a billion dollars on Nvidia GPUs, the surrounding infrastructure, cables, switches, transceivers, optical components has to be upgraded to match. The smarter the GPU gets, the more the interconnect matters. Nvidia has committed at least $6.5 billion to photonics companies in the past 4 months alone and the companies building the roads between the GPUs may end up being just as valuable as the companies building the GPUs themselves. Follow me Melvin for more AI, semis and the next big market themes.

Jensen Huang is investing in every photonics company he can find and the reason why tells you everything about where AI is headed (Save this). Lip-Bu Tan, the CEO of Intel says, when he looks for investment opportunities, he looks for the bottleneck and right now, the bottleneck is the interconnect, the pipes that move data between chips inside an AI data center. That is why he backed Credo Semiconductor, Astera Labs and Celestial AI on the optical side. Here is the simple version of what the interconnect bottleneck actually means. Think of an AI data center like a city, the GPUs are the buildings where all the work happens but for those buildings to function, you need roads connecting them, fast roads that can carry enormous traffic without congestion. And those roads are now the single biggest constraint on AI performance. As clusters scale to hundreds of thousands of GPUs, traditional copper wiring is hitting its physical limits and that is where this entire sector comes in. Credo Semiconductor (CRDO) is the most direct pure play on this theme, Credo makes high speed cables and optical chips that connect GPUs inside data center racks. Their revenue tripled in fiscal 2026 to $1.3 billion, growing 272% year over year at its peak and four of the world's largest hyperscalers each individually account for more than 10% of Credo's revenue. Astera Labs (ALAB) solves the connection problem between different chip types. Astera makes the PCIe and connectivity chips that manage data flow between GPUs, CPUs, and memory without errors or slowdowns. Their revenue grew 93% year over year to $308 million in Q1 2026 alone. The optical companies are where the longer-term and potentially larger opportunity lives. Copper has physical limits, you can only push electrical signals so far before the signal degrades, the heat spikes and power consumption explodes. The solution is light, fiber optic connections that move data using photons instead of electrons which is faster, cooler and far more energy efficient. Jensen Huang made this clear at Computex 2026 because copper works as long as physically possible but at greater distances and larger scale, optics takes over. Coherent (COHR) is the most established optical company in this space. Coherent makes the lasers, transceivers, and optical components at the foundation of all fiber optic communications. Nvidia signed a multibillion-dollar purchase commitment and invested $2 billion directly into the company and their customer order books are already extending out to 2028. Marvell (MRVL) is the most comprehensive bet across the entire connectivity stack. Marvell makes chips for optical networking, PCIe switching and custom AI silicon. Jensen Huang called Marvell the next trillion dollar company at Computex 2026 and backed it with a $2 billion Nvidia investment. Marvell also acquired Celestial AI, the exact company Lip-Bu Tan backed for $3.25 billion, gaining photonic fabric technology delivering 16 terabits per second of bandwidth. Lumentum (LITE), Corning (GLW), and Ciena (CIEN) round out the major public names. Lumentum received a $2 billion Nvidia investment for laser and photonics components. Corning known mostly for phone glass received $500 million from Nvidia for optical connectivity work and is up over 100% year to date. Ciena runs the optical networking systems between data centers and is seeing analyst price targets raised on the back of the AI optics boom. Every time a hyperscaler spends a billion dollars on Nvidia GPUs, the surrounding infrastructure, cables, switches, transceivers, optical components has to be upgraded to match. The smarter the GPU gets, the more the interconnect matters. Nvidia has committed at least $6.5 billion to photonics companies in the past 4 months alone and the companies building the roads between the GPUs may end up being just as valuable as the companies building the GPUs themselves. Follow me Melvin for more AI, semis and the next big market themes.

Melvin

151,246 просмотров • 16 дней назад

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution paper page: Text-based diffusion models have exhibited remarkable success in generation and editing, showing great promise for enhancing visual content with their generative prior. However, applying these models to video super-resolution remains challenging due to the high demands for output fidelity and temporal consistency, which is complicated by the inherent randomness in diffusion models. Our study introduces Upscale-A-Video, a text-guided latent diffusion framework for video upscaling. This framework ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences; globally, without training, a flow-guided recurrent latent propagation module is introduced to enhance overall video stability by propagating and fusing latent across the entire sequences. Thanks to the diffusion paradigm, our model also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation, enabling a trade-off between fidelity and quality. Extensive experiments show that Upscale-A-Video surpasses existing methods in both synthetic and real-world benchmarks, as well as in AI-generated videos, showcasing impressive visual realism and temporal consistency.

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution paper page: Text-based diffusion models have exhibited remarkable success in generation and editing, showing great promise for enhancing visual content with their generative prior. However, applying these models to video super-resolution remains challenging due to the high demands for output fidelity and temporal consistency, which is complicated by the inherent randomness in diffusion models. Our study introduces Upscale-A-Video, a text-guided latent diffusion framework for video upscaling. This framework ensures temporal coherence through two key mechanisms: locally, it integrates temporal layers into U-Net and VAE-Decoder, maintaining consistency within short sequences; globally, without training, a flow-guided recurrent latent propagation module is introduced to enhance overall video stability by propagating and fusing latent across the entire sequences. Thanks to the diffusion paradigm, our model also offers greater flexibility by allowing text prompts to guide texture creation and adjustable noise levels to balance restoration and generation, enabling a trade-off between fidelity and quality. Extensive experiments show that Upscale-A-Video surpasses existing methods in both synthetic and real-world benchmarks, as well as in AI-generated videos, showcasing impressive visual realism and temporal consistency.

AK

32,840 просмотров • 2 лет назад

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Santiago

164,162 просмотров • 2 лет назад

Tune Studio is an end-to-end platform for developing applications using Large Language Models. So far, I haven't seen any other platform like this one. You can do everything here: 1. You can curate your data. 2. Use the playground to play with different models and try your ideas. 3. Fine-tune an open-source model on your data. 4. Deploy the model when you are done. This is awesome for anyone building generative AI applications. You can use Tune Studio to work with any of the open-source models out there. They were one of the few companies to host Llama 2 and Llama 3 before anyone else. Here is a link to check it out: One of their main selling points is that Tune Studio scales! You don't have to worry about serving your model to lots of users. They also have built-in user management, authentication, on-prem support, user context management, and pretty much everything you need to build generative AI applications. Thanks to the Tune team for collaborating with me on this post. We are living through the best years of development tools for AI developers. The field is unstoppable.

Tune Studio is an end-to-end platform for developing applications using Large Language Models. So far, I haven't seen any other platform like this one. You can do everything here: 1. You can curate your data. 2. Use the playground to play with different models and try your ideas. 3. Fine-tune an open-source model on your data. 4. Deploy the model when you are done. This is awesome for anyone building generative AI applications. You can use Tune Studio to work with any of the open-source models out there. They were one of the few companies to host Llama 2 and Llama 3 before anyone else. Here is a link to check it out: One of their main selling points is that Tune Studio scales! You don't have to worry about serving your model to lots of users. They also have built-in user management, authentication, on-prem support, user context management, and pretty much everything you need to build generative AI applications. Thanks to the Tune team for collaborating with me on this post. We are living through the best years of development tools for AI developers. The field is unstoppable.

Santiago

39,101 просмотров • 2 лет назад

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

🍺 LagerNVS (CVPR 2026) 🍺 LagerNVS is a generalizable, feed-forward, real-time Novel View Synthesis network which - performs rendering in real time, - generalizes to in-the-wild data, - works with and without known source cameras, - sets a new state-of-the-art among deterministic methods, - can be paired with a diffusion decoder for generative extrapolation. LagerNVS shows that 3D biases are useful for Novel View Synthesis but explicit 3D representations are not required to achieve them. We use 3D biases in (1) architecture design and (2) pre-training: (1) In NVS with explicit 3D representations (3DGS, NeRF) reconstruction is typically difficult and slow, but rendering is much faster and simpler. We mimic this process in the network design: we use a large (1B params) encoder and a small, lightweight decoder (ViT-B). This allows increasing the network capacity while still achieving real-time rendering. (2) The encoder, initialized from VGGT, was pre-trained with 3D reconstruction objectives, making the initial features 3D aware. Both substantially improve performance. Project page: Code: Paper: Models: Work done with Jianyuan Minghao Chen Christian Rupprecht and Andrea Vedaldi

Stan Szymanowicz

31,570 просмотров • 3 месяцев назад

Today we’re releasing V-JEPA, a method for teaching machines to understand and model the physical world by watching videos. This work is another important step towards Yann LeCun’s outlined vision of AI models that use a learned understanding of the world to plan, reason and accomplish complex tasks. Details ➡️ We're releasing a collection of V-JEPA vision models trained with a feature prediction objective using self-supervised learning. The models are able to understand and predict what is going on in a video, even with limited information. It learns by predicting missing or obscured parts of a video in its internal feature space. Unlike generative approaches that fill in missing pixels, this flexible approach enables up to 6x improvements in training and sample efficiency. The models were pre-trained on entirely unlabeled data, and a small amount of labeled data can be used to train a task-specific prediction head on top after pre-training. Our results show that, using a frozen backbone, our top V-JEPA models achieve 82.0% on Kinetics-400, 72.2% on Something-Something-v2 and 77.9% on ImageNet1K — competitive with or exceeding previous leading video models. We believe that this work is an important milestone on the path to advancing machine intelligence.

Today we’re releasing V-JEPA, a method for teaching machines to understand and model the physical world by watching videos. This work is another important step towards Yann LeCun’s outlined vision of AI models that use a learned understanding of the world to plan, reason and accomplish complex tasks. Details ➡️ We're releasing a collection of V-JEPA vision models trained with a feature prediction objective using self-supervised learning. The models are able to understand and predict what is going on in a video, even with limited information. It learns by predicting missing or obscured parts of a video in its internal feature space. Unlike generative approaches that fill in missing pixels, this flexible approach enables up to 6x improvements in training and sample efficiency. The models were pre-trained on entirely unlabeled data, and a small amount of labeled data can be used to train a task-specific prediction head on top after pre-training. Our results show that, using a frozen backbone, our top V-JEPA models achieve 82.0% on Kinetics-400, 72.2% on Something-Something-v2 and 77.9% on ImageNet1K — competitive with or exceeding previous leading video models. We believe that this work is an important milestone on the path to advancing machine intelligence.

AI at Meta

703,647 просмотров • 2 лет назад

Introducing open source, real-time demo for Comfy Deploy, streaming in any params (text, number, images). Workflow by Julien Blanchon 🇺🇦 🫶 Running serverless gpu on Modal Drawing UI tldraw Inspired by Krea, fal 's real-time demo! Huge respects! Demo on GitHub. For context: tbh this is just a technical demo, truly meant to showcase how powerfully Comfy UI is when you get to expose the params. I'm also very grateful to even get the chance to meet with the team behind krea and fal in SF, these are very creative people and I have huge respect. Everything you see here is running behind a workflow done by julien when I asked for help in the discord, for which I am really really grateful. I'm looking toward a future where real-time generative AI becomes more accessible, and by contributing to open source, a lot more people can build creative AI apps and let their imaginations go wild. benny

Introducing open source, real-time demo for Comfy Deploy, streaming in any params (text, number, images). Workflow by Julien Blanchon 🇺🇦 🫶 Running serverless gpu on Modal Drawing UI tldraw Inspired by Krea, fal 's real-time demo! Huge respects! Demo on GitHub. For context: tbh this is just a technical demo, truly meant to showcase how powerfully Comfy UI is when you get to expose the params. I'm also very grateful to even get the chance to meet with the team behind krea and fal in SF, these are very creative people and I have huge respect. Everything you see here is running behind a workflow done by julien when I asked for help in the discord, for which I am really really grateful. I'm looking toward a future where real-time generative AI becomes more accessible, and by contributing to open source, a lot more people can build creative AI apps and let their imaginations go wild. benny

BennyKok

165,435 просмотров • 2 лет назад

VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored. TimSong, Long Le, and our GRASP Laboratory team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time. We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies. This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence. Project: Paper:

VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored. TimSong, Long Le, and our GRASP Laboratory team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time. We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies. This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence. Project: Paper:

Kostas Daniilidis

47,593 просмотров • 4 месяцев назад

Today Mirror Mirror AI is launching the marketplace where fashion models license their likeness and brands get stunning AI-generated imagery featuring real people. Commercially licensed, model-approved. Try our platform: As a fashion model I used to spend hours on fashion photoshoot sets. I later did my PhD in CS and became a Research Scientist on AI for fashion. I can see clearly that AI image generation is replacing a large portion of my old job. But brands that use AI recklessly have already paid the price. It damages reputations and hurts the bottom line. Putting real people at the core of AI-generated imagery isn't just about avoiding backlash. It's better business. That's what Mirror Mirror AI is built for. Right now, Mirror Mirror AI houses agency-signed models who have graced the covers of Vogue and Harper's Bazaar. You can digitally book them using our fashion-centric AI software, get your campaign done in hours instead of weeks, and never have to fly anyone in. You purchase a license for commercial use upon approval, and the models get paid. Mirror Mirror AI is also opening a global call for independent models from anywhere in the world to apply to be featured on the platform. Work with fashion brands internationally, choose the projects you take on, and earn from your own likeness on your own terms. Selected models will be announced at an exclusive event in New York during Tech Week this June. Apply for the open call: A huge thank you to our incredible team for pouring their hearts into this launch, and to a16z a16z speedrun 🧊 for believing in our vision from the start. We're just getting started.

Today Mirror Mirror AI is launching the marketplace where fashion models license their likeness and brands get stunning AI-generated imagery featuring real people. Commercially licensed, model-approved. Try our platform: As a fashion model I used to spend hours on fashion photoshoot sets. I later did my PhD in CS and became a Research Scientist on AI for fashion. I can see clearly that AI image generation is replacing a large portion of my old job. But brands that use AI recklessly have already paid the price. It damages reputations and hurts the bottom line. Putting real people at the core of AI-generated imagery isn't just about avoiding backlash. It's better business. That's what Mirror Mirror AI is built for. Right now, Mirror Mirror AI houses agency-signed models who have graced the covers of Vogue and Harper's Bazaar. You can digitally book them using our fashion-centric AI software, get your campaign done in hours instead of weeks, and never have to fly anyone in. You purchase a license for commercial use upon approval, and the models get paid. Mirror Mirror AI is also opening a global call for independent models from anywhere in the world to apply to be featured on the platform. Work with fashion brands internationally, choose the projects you take on, and earn from your own likeness on your own terms. Selected models will be announced at an exclusive event in New York during Tech Week this June. Apply for the open call: A huge thank you to our incredible team for pouring their hearts into this launch, and to a16z a16z speedrun 🧊 for believing in our vision from the start. We're just getting started.

Yusan Lin

234,364 просмотров • 3 месяцев назад

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

AK

429,307 просмотров • 3 лет назад

New episode of the Information Bottleneck! We talked with Stefano Ermon about why he thinks diffusion LLMs will replace autoregressive ones. Stefano co-invented DDIM, FlashAttention, DPO, and score-based diffusion models. He's a Stanford professor and now runs Inception AI, where they built Mercury II. We go deep but also cover the bigger picture - the startup journey, PhD vs industry, and where AI is heading. A few things that stuck with me: - He thinks of autoregressive models as typewriters and diffusion models as editors. One goes left to right. The other starts messy and refines. - Mercury II (their text difussion model) already beats the fastest autoregressive models on latency-critical stuff as voice agents, code suggestions, anything where you have a tight time budget. And it does it because diffusion generates tokens in parallel instead of one at a time. - We also got into whether AI will actually replace software engineers (his answer: no), PhD vs industry advice, and what it was like going from an ICML best paper to raising money.

New episode of the Information Bottleneck! We talked with Stefano Ermon about why he thinks diffusion LLMs will replace autoregressive ones. Stefano co-invented DDIM, FlashAttention, DPO, and score-based diffusion models. He's a Stanford professor and now runs Inception AI, where they built Mercury II. We go deep but also cover the bigger picture - the startup journey, PhD vs industry, and where AI is heading. A few things that stuck with me: - He thinks of autoregressive models as typewriters and diffusion models as editors. One goes left to right. The other starts messy and refines. - Mercury II (their text difussion model) already beats the fastest autoregressive models on latency-critical stuff as voice agents, code suggestions, anything where you have a tight time budget. And it does it because diffusion generates tokens in parallel instead of one at a time. - We also got into whether AI will actually replace software engineers (his answer: no), PhD vs industry advice, and what it was like going from an ICML best paper to raising money.

Ravid Shwartz Ziv

22,350 просмотров • 3 месяцев назад

📢📢 𝐀𝐯𝐚𝐭𝟑𝐫 📢📢 Avat3r creates high-quality 3D head avatars from just a few input images in a single forward pass with a new dynamic 3DGS reconstruction model. Video: Project: Our core idea is to make Gaussian Reconstruction Models animatable. We find that a simple cross-attention to an expression code sequence is already sufficient to model complex facial expressions. We then incorporate position maps from DUSt3R and feature maps from Sapiens to facilitate the prediction task. While DUSt3R's position maps act as a pixel-aligned initialization for the Gaussians' positions, the Sapiens feature maps help the cross-view transformer to match corresponding image tokens in the 4 input images. One major challenge in creating a 3D head avatar from smartphone images comes from inconsistent facial expressions when the subject could not remain perfectly static during the capture. We eliminate this static requirement by simply showing our model input images with different facial expressions during training. This technique makes our model robust to inconsistent input images later on. Finally, we show that despite the model has been trained with 4 input images, one can even create a 3D head avatar when only a single image is available. To achieve this, we employ a pre-trained 3D GAN to lift the single image to 3D and then render the 4 input images for our model. This allows us to create 3D head avatars from single images and even highly out-of-distribution examples like AI generated faces, paintings or statues. Great work by Tobias Kirschstein from his internship at Meta with Javier Romero, Artem Sevastopolsky, and Shunsuke Saito

Matthias Niessner

74,698 просмотров • 1 год назад

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress continuous-time trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Davide Scaramuzza

12,637 просмотров • 2 лет назад

InstantDrag Improving Interactivity in Drag-based Image Editing discuss: Drag-based image editing has recently gained popularity for its interactivity and precision. However, despite the ability of text-to-image models to generate samples within a second, drag editing still lags behind due to the challenge of accurately reflecting user interaction while maintaining image content. Some existing approaches rely on computationally intensive per-image optimization or intricate guidance-based methods, requiring additional inputs such as masks for movable regions and text prompts, thereby compromising the interactivity of the editing process. We introduce InstantDrag, an optimization-free pipeline that enhances interactivity and speed, requiring only an image and a drag instruction as input. InstantDrag consists of two carefully designed networks: a drag-conditioned optical flow generator (FlowGen) and an optical flow-conditioned diffusion model (FlowDiffusion). InstantDrag learns motion dynamics for drag-based image editing in real-world video datasets by decomposing the task into motion generation and motion-conditioned image generation. We demonstrate InstantDrag's capability to perform fast, photo-realistic edits without masks or text prompts through experiments on facial video datasets and general scenes. These results highlight the efficiency of our approach in handling drag-based image editing, making it a promising solution for interactive, real-time applications.

InstantDrag Improving Interactivity in Drag-based Image Editing discuss: Drag-based image editing has recently gained popularity for its interactivity and precision. However, despite the ability of text-to-image models to generate samples within a second, drag editing still lags behind due to the challenge of accurately reflecting user interaction while maintaining image content. Some existing approaches rely on computationally intensive per-image optimization or intricate guidance-based methods, requiring additional inputs such as masks for movable regions and text prompts, thereby compromising the interactivity of the editing process. We introduce InstantDrag, an optimization-free pipeline that enhances interactivity and speed, requiring only an image and a drag instruction as input. InstantDrag consists of two carefully designed networks: a drag-conditioned optical flow generator (FlowGen) and an optical flow-conditioned diffusion model (FlowDiffusion). InstantDrag learns motion dynamics for drag-based image editing in real-world video datasets by decomposing the task into motion generation and motion-conditioned image generation. We demonstrate InstantDrag's capability to perform fast, photo-realistic edits without masks or text prompts through experiments on facial video datasets and general scenes. These results highlight the efficiency of our approach in handling drag-based image editing, making it a promising solution for interactive, real-time applications.

AK

71,232 просмотров • 1 год назад

Today, we’re excited to announce our $50M Series B, led by Greenfield Partners, with participation from Lightspeed and Notable Capital. 🚀 At Patronus AI, we develop simulations and evals to train and improve AI. The first phase of AI was built on static benchmarks, but that era is over. As agents are used to solve longer and longer tasks, they need to practice in dynamic, living worlds to get better. Simulations are the critical infrastructure powering this next phase. As a company, we’re behind the most influential research and products in AI evaluation, like FinanceBench, Lynx, and Percival. And things have moved at the speed of light since.⚡ We partner with the world's leading frontier AI labs and enterprises, and our revenue has grown more than 15x over the past year. Additionally, today, we’re introducing a preview of the first Digital World Model for AI agent training and simulation: Patronus-DWM. Digital World Models are language diffusion world models that predict realistic environment behaviors and steer agent actions across digital workflows. Just as physical world models predict how objects move through space, we’re developing the equivalent for the digital world: predicting how agents act in digital workflows, then using that to scale the creation of high-quality training data for LLMs. Digital World Models help us push the frontier of ultra long horizon workflows, and unlock a new class of self-improving RL environments. This is our scalable approach to simulating all of the world’s intelligence. The round was also joined by Datadog, Inc., Samsung Ventures, Gokul Rajaram, Factorial Capital, and a large cohort of amazing AI leaders across Anthropic, OpenAI, Google DeepMind, NVIDIA, Recursive, and more.✨ It has been the ride of a lifetime. But we’re just getting started. The best is yet to come. "Do not go gentle into that good night, Rage, rage against the dying of the light" - Dylan Thomas (1954)

Today, we’re excited to announce our $50M Series B, led by Greenfield Partners, with participation from Lightspeed and Notable Capital. 🚀 At Patronus AI, we develop simulations and evals to train and improve AI. The first phase of AI was built on static benchmarks, but that era is over. As agents are used to solve longer and longer tasks, they need to practice in dynamic, living worlds to get better. Simulations are the critical infrastructure powering this next phase. As a company, we’re behind the most influential research and products in AI evaluation, like FinanceBench, Lynx, and Percival. And things have moved at the speed of light since.⚡ We partner with the world's leading frontier AI labs and enterprises, and our revenue has grown more than 15x over the past year. Additionally, today, we’re introducing a preview of the first Digital World Model for AI agent training and simulation: Patronus-DWM. Digital World Models are language diffusion world models that predict realistic environment behaviors and steer agent actions across digital workflows. Just as physical world models predict how objects move through space, we’re developing the equivalent for the digital world: predicting how agents act in digital workflows, then using that to scale the creation of high-quality training data for LLMs. Digital World Models help us push the frontier of ultra long horizon workflows, and unlock a new class of self-improving RL environments. This is our scalable approach to simulating all of the world’s intelligence. The round was also joined by Datadog, Inc., Samsung Ventures, Gokul Rajaram, Factorial Capital, and a large cohort of amazing AI leaders across Anthropic, OpenAI, Google DeepMind, NVIDIA, Recursive, and more.✨ It has been the ride of a lifetime. But we’re just getting started. The best is yet to come. "Do not go gentle into that good night, Rage, rage against the dying of the light" - Dylan Thomas (1954)

PatronusAI

93,885 просмотров • 19 дней назад

this is huge news from Cursor, they've pulled of the impossible and turned their ai-wrapper into a in-destructible moat sam altman literally called it cursor's ai agent harness is available for anyone to build on, which means ai models are now a commodity > 1-time install and now anyone can run cursor's agent locally or via cloud. use any model (e.g. gpt 5.5) but with the added cursor harness that makes it 10X better. > its so good that 3 of cursors biggest competitors are embedding it into their products. > now cursor DOESN'T DEPEND on anthropic or openai. their own model (composer 2) competes directly! yesterday sam altman said the ai model and harness are one and the same and today cursor turned their harness into a self-owned moat fucking masterclass (coming from a former cursor / ai wrapper hater)

this is huge news from Cursor, they've pulled of the impossible and turned their ai-wrapper into a in-destructible moat sam altman literally called it cursor's ai agent harness is available for anyone to build on, which means ai models are now a commodity > 1-time install and now anyone can run cursor's agent locally or via cloud. use any model (e.g. gpt 5.5) but with the added cursor harness that makes it 10X better. > its so good that 3 of cursors biggest competitors are embedding it into their products. > now cursor DOESN'T DEPEND on anthropic or openai. their own model (composer 2) competes directly! yesterday sam altman said the ai model and harness are one and the same and today cursor turned their harness into a self-owned moat fucking masterclass (coming from a former cursor / ai wrapper hater)

Ejaaz

949,736 просмотров • 2 месяцев назад

Today, we’re excited to announce our $50M Series B, led by Greenfield Partners (formerly TPG Capital), with participation from Lightspeed and Notable Capital. 🚀 At PatronusAI, we develop simulations and evals to train and improve AI. The first phase of AI was built on static benchmarks, but that era is over now. As agents are used to solve longer and longer tasks, they need to practice in dynamic, living worlds to get better. Simulations are the critical infrastructure powering this next phase. As a company, we’re behind the most influential research and products in AI evaluation, like FinanceBench, Lynx, and Percival. And things have moved at the speed of light since. ⚡ We partner with the world's leading frontier AI labs and enterprises, and our revenue has grown more than 15x over the past year. Additionally, today, we’re introducing a preview of the first Digital World Model for AI agent training and simulation: Patronus-DWM. Digital World Models are language diffusion world models that predict realistic environment behaviors and steer agent actions across digital workflows. Just as physical world models predict how objects move through space, we’re developing the equivalent for the digital world: predicting how agents act in digital workflows, then using that to scale the creation of high-quality training data for LLMs. Digital World Models help us push the frontier of ultra long horizon workflows, and unlock a new class of self-improving RL environments. This is our scalable approach to simulating all of the world’s intelligence. The round was also joined by Datadog, Inc., Samsung Ventures, Gokul Rajaram, Factorial Capital, and a large cohort of amazing AI leaders and researchers across Anthropic, OpenAI, Google DeepMind, NVIDIA, Recursive, and more. ✨ It has been the ride of a lifetime. But we’re just getting started. The best is yet to come. "Do not go gentle into that good night, Rage, rage against the dying of the light" - Dylan Thomas (1954)

Today, we’re excited to announce our $50M Series B, led by Greenfield Partners (formerly TPG Capital), with participation from Lightspeed and Notable Capital. 🚀 At PatronusAI, we develop simulations and evals to train and improve AI. The first phase of AI was built on static benchmarks, but that era is over now. As agents are used to solve longer and longer tasks, they need to practice in dynamic, living worlds to get better. Simulations are the critical infrastructure powering this next phase. As a company, we’re behind the most influential research and products in AI evaluation, like FinanceBench, Lynx, and Percival. And things have moved at the speed of light since. ⚡ We partner with the world's leading frontier AI labs and enterprises, and our revenue has grown more than 15x over the past year. Additionally, today, we’re introducing a preview of the first Digital World Model for AI agent training and simulation: Patronus-DWM. Digital World Models are language diffusion world models that predict realistic environment behaviors and steer agent actions across digital workflows. Just as physical world models predict how objects move through space, we’re developing the equivalent for the digital world: predicting how agents act in digital workflows, then using that to scale the creation of high-quality training data for LLMs. Digital World Models help us push the frontier of ultra long horizon workflows, and unlock a new class of self-improving RL environments. This is our scalable approach to simulating all of the world’s intelligence. The round was also joined by Datadog, Inc., Samsung Ventures, Gokul Rajaram, Factorial Capital, and a large cohort of amazing AI leaders and researchers across Anthropic, OpenAI, Google DeepMind, NVIDIA, Recursive, and more. ✨ It has been the ride of a lifetime. But we’re just getting started. The best is yet to come. "Do not go gentle into that good night, Rage, rage against the dying of the light" - Dylan Thomas (1954)

Anand Kannappan

38,520 просмотров • 19 дней назад

🚨 Paper Alert 🚨 ➡️Paper Title: Articulate3D: Zero-Shot Text-Driven 3D Object Posing 🌟Few pointers from the paper 🎯Authors of this paper proposed a training-free method, “Articulate3D”, to pose a 3D asset through language control. 🎯Despite advances in vision and language models, this task remains surprisingly challenging. 🎯To achieve this goal, they decomposed the problem into two steps. 🎯They modified a powerful image-generator to create target images conditioned on the input image and a text instruction. 🎯They then align the mesh to the target images through a multi-view pose optimisation step. 🎯 In detail, they introduced a self-attention rewiring mechanism (RSActrl) that decouples the source structure from pose within an image generative model, allowing it to maintain a consistent structure across varying poses. 🎯They observed that differentiable rendering is an unreliable signal for articulation optimisation; instead, they used keypoints to establish correspondences between input and target images. 🎯The effectiveness of Articulate3D is demonstrated across a diverse range of 3D objects and free-form text prompts, successfully manipulating poses while maintaining the original identity of the mesh. 🎯Quantitative evaluations and a comparative user study, in which their method was preferred over 85% of the time, confirm its superiority over existing approaches. 🏢Organization: University of Oxford , Google DeepMind 🧙Paper Authors: Oishi Deb, Anjun Hu, Ashkan Khakzar, Philip Torr, Christian Rupprecht 📝 Read the Full Paper here: 🗂️ Project Page: 🎥 Be sure to watch the attached Demo Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️QT and teach your network something new Follow me 👣, naveen manwani , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements.

🚨 Paper Alert 🚨 ➡️Paper Title: Articulate3D: Zero-Shot Text-Driven 3D Object Posing 🌟Few pointers from the paper 🎯Authors of this paper proposed a training-free method, “Articulate3D”, to pose a 3D asset through language control. 🎯Despite advances in vision and language models, this task remains surprisingly challenging. 🎯To achieve this goal, they decomposed the problem into two steps. 🎯They modified a powerful image-generator to create target images conditioned on the input image and a text instruction. 🎯They then align the mesh to the target images through a multi-view pose optimisation step. 🎯 In detail, they introduced a self-attention rewiring mechanism (RSActrl) that decouples the source structure from pose within an image generative model, allowing it to maintain a consistent structure across varying poses. 🎯They observed that differentiable rendering is an unreliable signal for articulation optimisation; instead, they used keypoints to establish correspondences between input and target images. 🎯The effectiveness of Articulate3D is demonstrated across a diverse range of 3D objects and free-form text prompts, successfully manipulating poses while maintaining the original identity of the mesh. 🎯Quantitative evaluations and a comparative user study, in which their method was preferred over 85% of the time, confirm its superiority over existing approaches. 🏢Organization: University of Oxford , Google DeepMind 🧙Paper Authors: Oishi Deb, Anjun Hu, Ashkan Khakzar, Philip Torr, Christian Rupprecht 📝 Read the Full Paper here: 🗂️ Project Page: 🎥 Be sure to watch the attached Demo Video - Sound on 🔊🔊 Find this Valuable 💎 ? ♻️QT and teach your network something new Follow me 👣, naveen manwani , for the latest updates on Tech and AI-related news, insightful research papers, and exciting announcements.

naveen manwani

14,334 просмотров • 10 месяцев назад