Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

1/8 Does text-to-image generation need language pretraining? Today, we share Halcyon, a family of early-fusion, base models trained from scratch. Halcyon outperforms strong late-fusion baselines with up to 275× less pretraining compute. Models and recipes are open. Halcyon is part of a research program at Radical Numerics on the... show more

Radical Numerics

4,434 subscribers

76,603 görüntüleme • 19 gün önce •via X (Twitter)

Eğitim Bilim & Teknoloji

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Switti -- a new scale-wise transformer for text-to-image generation 🦾 🔥 Improved generation of fine-grained details. Outperforms existing T2I AR models and competes with state-of-the-art T2I diffusion models while being up to 7x faster.

Switti -- a new scale-wise transformer for text-to-image generation 🦾 🔥 Improved generation of fine-grained details. Outperforms existing T2I AR models and competes with state-of-the-art T2I diffusion models while being up to 7x faster.

Gradio

29,314 görüntüleme • 1 yıl önce

STEVE-1: A Generative Model for Text-to-Behavior in Minecraft paper page: Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces an instruction-tuned Video Pretraining (VPT) model for Minecraft called STEVE-1, demonstrating that the unCLIP approach, utilized in DALL-E 2, is also effective for creating instruction-following sequential decision-making agents. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, bypassing the need for costly human text annotations. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 costs just $60 to train and can follow a wide range of short-horizon open-ended text and visual instructions in Minecraft. STEVE-1 sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.

STEVE-1: A Generative Model for Text-to-Behavior in Minecraft paper page: Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces an instruction-tuned Video Pretraining (VPT) model for Minecraft called STEVE-1, demonstrating that the unCLIP approach, utilized in DALL-E 2, is also effective for creating instruction-following sequential decision-making agents. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, bypassing the need for costly human text annotations. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 costs just $60 to train and can follow a wide range of short-horizon open-ended text and visual instructions in Minecraft. STEVE-1 sets a new bar for open-ended instruction following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.

AK

144,827 görüntüleme • 3 yıl önce

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵

Amir Zamir

24,521 görüntüleme • 3 ay önce

Pokémon Mystery Dungeon: Halcyon is available now! Recapturing the vibes of PMD Explorers of Sky! In Halcyon, you've turned into a Pokémon! A brand new story, meet a fresh new cast of pals and go on amazing adventures! Check out + Palika #EeveeExpo3

Eevee Expo

17,126 görüntüleme • 2 yıl önce

Exciting news from Jasper Research! 🚀 We built an internal 4B text-to-image model from scratch that turned out to be one of the fastest models out there! 1k image generation in less than a second! Our technical report shares everything we learned along the way. 👇

Exciting news from Jasper Research! 🚀 We built an internal 4B text-to-image model from scratch that turned out to be one of the fastest models out there! 1k image generation in less than a second! Our technical report shares everything we learned along the way. 👇

Damien Henry

14,960 görüntüleme • 2 ay önce

1/ With Omnii Diagnostics, we asked whether a multimodal genome language model could help find cancers whose signals in blood are too faint for current classifiers. We adapted Omnii-1 to a biomarker never seen in pretraining. In a test cohort, Omnii's features improved early-stage cancer detection

1/ With Omnii Diagnostics, we asked whether a multimodal genome language model could help find cancers whose signals in blood are too faint for current classifiers. We adapted Omnii-1 to a biomarker never seen in pretraining. In a test cohort, Omnii's features improved early-stage cancer detection

Radical Numerics

17,686 görüntüleme • 6 gün önce

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢 We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive than classical 3DMMs such as FLAME. Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control. Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity. Key to successful and generalized training of our ViT-based network are: (1) large-scale registration of existing 3D head datasets, and (2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals. Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization. 🌍 🎥 Great work by Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen

📢Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image📢 We directly regress neural parametric head models (NPHMs) from a single image — fast, stable, and significantly more expressive than classical 3DMMs such as FLAME. Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control. Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity. Key to successful and generalized training of our ViT-based network are: (1) large-scale registration of existing 3D head datasets, and (2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals. Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization. 🌍 🎥 Great work by Simon Giebenhain, Tobias Kirschstein, Liam Schoneveld, Davide Davoli, Zhe Chen

Matthias Niessner

37,993 görüntüleme • 10 ay önce

Massive update for AI Engineers! Training diffusion models just got a lot easier. dLLM is an open-source library that does for diffusion models what Hugging Face did for transformers. Here's why this matters: Traditional autoregressive models generate text left-to-right, one token at a time. Diffusion models work differently - they refine the entire sequence iteratively, giving you better control over generation quality and more flexible editing capabilities. The problem? Building and training these models required stitching together scattered tools and reimplementing research papers from scratch. Most of the tooling has been scattered and is hard to reproduce. dLLM changes this: It unifies everything you need to train, evaluate, and deploy diffusion language models: ↳ Scalable training with LoRA, DeepSpeed, and FSDP support ↳ Unified evaluation that abstracts away inference complexity ↳ Ready-to-use recipes for pretraining, finetuning, and evaluation The library includes implementations of models like LLaDA and Dream, plus training algorithms like Edit Flows that enable insertion, deletion, and substitution operations. The team just released ModernBERT-Chat models showing you can turn BERT into lightweight chatbots through masked instruction tuning. This is practical and worth exploring. The setup is straightforward. Run locally with Accelerate or scale to multi-node clusters with Slurm. If you're working with language models and want to explore diffusion-based approaches without rebuilding infrastructure, dLLM gives you a production-ready starting point. Link to the repo in the next tweet.

Massive update for AI Engineers! Training diffusion models just got a lot easier. dLLM is an open-source library that does for diffusion models what Hugging Face did for transformers. Here's why this matters: Traditional autoregressive models generate text left-to-right, one token at a time. Diffusion models work differently - they refine the entire sequence iteratively, giving you better control over generation quality and more flexible editing capabilities. The problem? Building and training these models required stitching together scattered tools and reimplementing research papers from scratch. Most of the tooling has been scattered and is hard to reproduce. dLLM changes this: It unifies everything you need to train, evaluate, and deploy diffusion language models: ↳ Scalable training with LoRA, DeepSpeed, and FSDP support ↳ Unified evaluation that abstracts away inference complexity ↳ Ready-to-use recipes for pretraining, finetuning, and evaluation The library includes implementations of models like LLaDA and Dream, plus training algorithms like Edit Flows that enable insertion, deletion, and substitution operations. The team just released ModernBERT-Chat models showing you can turn BERT into lightweight chatbots through masked instruction tuning. This is practical and worth exploring. The setup is straightforward. Run locally with Accelerate or scale to multi-node clusters with Slurm. If you're working with language models and want to explore diffusion-based approaches without rebuilding infrastructure, dLLM gives you a production-ready starting point. Link to the repo in the next tweet.

Akshay 🚀

64,011 görüntüleme • 11 ay önce

Introducing Muse Image and Muse Video, the first media generation models developed by Meta Superintelligence Labs. Muse Image is our most advanced image generation model yet. It follows instructions faithfully, edits with precision, composes from multiple references, and draws on Instagram for social context. It also brings agentic tool use capabilities to image generation and integrates with Muse Spark. You can try Muse Image in the Meta AI app and web, as well as in Instagram Stories and WhatsApp – starting in limited countries with more locations on the way. Today we’re also previewing Muse Video, which is built upon the same pretraining base as Muse Image to deliver exceptional visual fidelity with native audio support. Learn more about both models:

Introducing Muse Image and Muse Video, the first media generation models developed by Meta Superintelligence Labs. Muse Image is our most advanced image generation model yet. It follows instructions faithfully, edits with precision, composes from multiple references, and draws on Instagram for social context. It also brings agentic tool use capabilities to image generation and integrates with Muse Spark. You can try Muse Image in the Meta AI app and web, as well as in Instagram Stories and WhatsApp – starting in limited countries with more locations on the way. Today we’re also previewing Muse Video, which is built upon the same pretraining base as Muse Image to deliver exceptional visual fidelity with native audio support. Learn more about both models:

AI at Meta

863,210 görüntüleme • 3 ay önce

Show-o One Single Transformer to Unify Multimodal Understanding and Generation discuss: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model.

Show-o One Single Transformer to Unify Multimodal Understanding and Generation discuss: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model.

AK

124,085 görüntüleme • 2 yıl önce

Training Wan 2.1-1.3B to use Qwen3-VL-2B text encoder. Doing 33% text only, 33% VL only, and 33% both. Just pretraining the two text input linear layers for now. It is amazing how quickly a model can adapt to a different text encoder. This is 25,750 steps, BS of 10.

Training Wan 2.1-1.3B to use Qwen3-VL-2B text encoder. Doing 33% text only, 33% VL only, and 33% both. Just pretraining the two text input linear layers for now. It is amazing how quickly a model can adapt to a different text encoder. This is 25,750 steps, BS of 10.

Ostris

26,778 görüntüleme • 4 ay önce

Introducing ✨Aya Expanse ✨ – an open-weights state-of-art family of models to help close the language gap with AI. Aya Expanse is both global and local. Driven by a multi-year commitment to multilingual research.

Introducing ✨Aya Expanse ✨ – an open-weights state-of-art family of models to help close the language gap with AI. Aya Expanse is both global and local. Driven by a multi-year commitment to multilingual research.

Cohere Labs

239,950 görüntüleme • 2 yıl önce

Backprop has been the only credit assignment algorithm capable of training large neural nets. Introducing Dust, the first zeroth-order method to pretrain transformers to approach and sometimes even *exceed* backprop with large amounts of computation (bitter lesson!) - Dust is on the order of 1,000 to 10,000x more compute efficient than EGGROLL, the state-of-the-art ES method, for training transformers. - We chose pretraining transformers because it's arguably the hardest task possible for zeroth-order optimization. - Search in high dimensions is really misunderstood: zeroth-order usually gets better with overparameterization, not worse, i.e. at a fixed population size, larger models (up to 120x) reach a lower loss than smaller ones. - Backprop-like gradients emerge from a large population of activation perturbations, and the alignment holds up at every scale we tested, up to 1B tokens, which is critical for scaling. The core idea is a *virtual population*, which helps scale to large populations in transformers by perturbing activations in parallel. w/ Bishwas, serdarml, Akshay

Backprop has been the only credit assignment algorithm capable of training large neural nets. Introducing Dust, the first zeroth-order method to pretrain transformers to approach and sometimes even exceed backprop with large amounts of computation (bitter lesson!) - Dust is on the order of 1,000 to 10,000x more compute efficient than EGGROLL, the state-of-the-art ES method, for training transformers. - We chose pretraining transformers because it's arguably the hardest task possible for zeroth-order optimization. - Search in high dimensions is really misunderstood: zeroth-order usually gets better with overparameterization, not worse, i.e. at a fixed population size, larger models (up to 120x) reach a lower loss than smaller ones. - Backprop-like gradients emerge from a large population of activation perturbations, and the alignment holds up at every scale we tested, up to 1B tokens, which is critical for scaling. The core idea is a virtual population, which helps scale to large populations in transformers by perturbing activations in parallel. w/ Bishwas, serdarml, Akshay

Samip

282,016 görüntüleme • 29 gün önce

Purple Halcyon🥵

Sensitive content

Purple Halcyon🥵

AK

21,232 görüntüleme • 19 gün önce

Introducing benchmark tracking across all of arXiv With thousands of models and benchmarks out there, it's impossible to track what's actually state-of-the-art We now index results from millions of papers to surface which models are SOTA according to open source research!

Introducing benchmark tracking across all of arXiv With thousands of models and benchmarks out there, it's impossible to track what's actually state-of-the-art We now index results from millions of papers to surface which models are SOTA according to open source research!

alphaXiv

10,892 görüntüleme • 10 ay önce

STRESSED IN THE SALLOWLANDS 😅 💀 There are only a few hours left until #AgeOfUmbra: Sallowlands returns on YouTube, Twitch - and Zachery Renauldo is feeling the pressure! Uncover the mysteries of the Halcyon Domain's haunted desert at 7PM PT tonight! 🗡️

STRESSED IN THE SALLOWLANDS 😅 💀 There are only a few hours left until #AgeOfUmbra: Sallowlands returns on YouTube, Twitch - and Zachery Renauldo is feeling the pressure! Uncover the mysteries of the Halcyon Domain's haunted desert at 7PM PT tonight! 🗡️

Critical Role

33,672 görüntüleme • 3 ay önce

You already know who Halcyon is. So... what are you waiting for?

You already know who Halcyon is. So... what are you waiting for?

Halcyon Trader Funding

8,877,370 görüntüleme • 3 ay önce

HEAD BACK TO THE HALCYON DOMAIN - TONIGHT! 🏜️ 💀 You heard Game Master Matthew Mercer! Our shaken survivors recover from a riptide and a hang glider ride as we return to the Sallowlands for more #Daggerheart adventure TONIGHT at 7PM PT on YouTube, and Twitch! 🗡️

HEAD BACK TO THE HALCYON DOMAIN - TONIGHT! 🏜️ 💀 You heard Game Master Matthew Mercer! Our shaken survivors recover from a riptide and a hang glider ride as we return to the Sallowlands for more #Daggerheart adventure TONIGHT at 7PM PT on YouTube, and Twitch! 🗡️

Critical Role

44,860 görüntüleme • 3 ay önce

NEW RELEASE: Today we're releasing CortexMAE: a family of fMRI foundation models trained on 2.1K hours of open fMRI data. We're also releasing Brainmarks: an open benchmark suite for evaluating fMRI foundation models. Full paper is on arXiv (accepted to ICML 2026) A thread:

NEW RELEASE: Today we're releasing CortexMAE: a family of fMRI foundation models trained on 2.1K hours of open fMRI data. We're also releasing Brainmarks: an open benchmark suite for evaluating fMRI foundation models. Full paper is on arXiv (accepted to ICML 2026) A thread:

Sophont

48,667 görüntüleme • 5 ay önce

Purple Halcyon🥵🔥

Sensitive content

Purple Halcyon🥵🔥

AK

49,313 görüntüleme • 11 gün önce