Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Today, we're shipping MLX support for TADA, our open-source text-to-speech model, which means the entire pipeline (LLM, flow-matching, and decoder) can now run locally on any Apple Silicon device. We're seeing a 45% reduction in memory usage and a 10x speed-up when using it quantized. With these improvements, you... show more

Hume AI

23,214 subscribers

24,684 Aufrufe • vor 4 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

This week, we're excited to announce our partnership with ElevenLabs — making it easier than ever to create videos with studio-grade voices, in any language, directly in Captions. With this partnership, you can dub your videos into different languages, select a professional-quality voice to narrate your content, or clone your voice so you can create polished videos any time, anywhere. Try it out and stay tuned for a third partnership announcement next week!

This week, we're excited to announce our partnership with ElevenLabs — making it easier than ever to create videos with studio-grade voices, in any language, directly in Captions. With this partnership, you can dub your videos into different languages, select a professional-quality voice to narrate your content, or clone your voice so you can create polished videos any time, anywhere. Try it out and stay tuned for a third partnership announcement next week!

Mirage (fka Captions)

14,628 Aufrufe • vor 1 Jahr

Introducing Novo Launching today a new project I coded for myself in 1 weekend in May and decided to finish this week. Novo is a dead simple to-do app that lets you "Speech-To-Tasks", or paste a huge text and organize for you. You can customize the AI and make it organize in any criteria: - Auto-tag by category - Schedule some types of tasks to certain days - Prioritize based on your own rules Try it:

Introducing Novo Launching today a new project I coded for myself in 1 weekend in May and decided to finish this week. Novo is a dead simple to-do app that lets you "Speech-To-Tasks", or paste a huge text and organize for you. You can customize the AI and make it organize in any criteria: - Auto-tag by category - Schedule some types of tasks to certain days - Prioritize based on your own rules Try it:

Pedro

76,232 Aufrufe • vor 1 Jahr

Introducing: OpenGranola 🔥 I built an open source meeting copilot for macOS. It transcribes both sides of your call on-device, searches your own notes in real time, and hands you talking points right when the conversation needs them. No audio leaves your Mac. Point it at a folder of markdown files, pick any LLM through OpenRouter (Claude, GPT-4o, Gemini, Llama), and it just works. It's invisible to screen share too — nobody knows you have it. The whole thing is open source. Link below

Introducing: OpenGranola 🔥 I built an open source meeting copilot for macOS. It transcribes both sides of your call on-device, searches your own notes in real time, and hands you talking points right when the conversation needs them. No audio leaves your Mac. Point it at a folder of markdown files, pick any LLM through OpenRouter (Claude, GPT-4o, Gemini, Llama), and it just works. It's invisible to screen share too — nobody knows you have it. The whole thing is open source. Link below

yazin

293,145 Aufrufe • vor 5 Monaten

You don't need a GPU for fast studio grade voice cloning anymore. Qwen3 TTS (1.7B Q4_K_M) + mainline llama.cpp is officially the fastest way to generate zero shot voice clones using 100% pure CPU execution. Following up on my last post where we ran the Q8 model on a GPU, we just took local C++ voice synthesis a massive step further. The open source community quantized Alibaba's SOTA Qwen3 TTS model down to Q4_K_M GGUF, completely freeing local audio pipelines from dedicated graphics hardware. Here is the real world benchmark and hardware breakdown of running SOTA voice cloning on CPU: # Architecture & Model Setup Using Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf paired with the 8 bit multimodal projector (mmproj-Q8_0.gguf), llama.cpp executes the entire pipeline in pure C++. No PyTorch, no CUDA dependencies, and no VRAM bottlenecks. # Real-World Memory Footprint - Baseline RAM: 1.6 GB system idle. - Peak Generation RAM: 8 GB RAM during active voice synthesis. - Requirement: Any basic machine with at least 8 GB of system RAM can run this easily. # Real World CPU Benchmarks - Google Colab Free Tier (Throttled 2 Core CPU): Synthesizes a 5 sec studio quality audio clip (~8 words) in 45 seconds. - Modern Consumer CPU (Intel i5/i7 13th/14th Gen or AMD Ryzen 7000/9000): generation should drop to 5 to 20 seconds (nearly 1:1 real-time generation speed!). # Zero Shot Voice Cloning Quality Pass any 5 to 20 second .wav audio sample to the C++ engine using the --tts-speaker-file flag. It yields clean, natural sounding cloned speech with virtually zero quality loss compared to unquantized FP16 weights. To make testing seamless, I built an updated zero config Google Colab notebook. It pulls the official pre built llama.cpp CPU binaries (zero compilation time!) launches a live Gradio web app right in your browser. Record a 5 second clip from your mic (or drop a .mp3, .wav file), type text, and generate cloned audio on CPU. Native C++ audio models are making edge based, offline AI voice agents a reality. Links to the free Q4 CPU Colab notebook and the Q4_K_M GGUF HuggingFace repository are in the replies below! Which models have you been running on your CPUs? What CPU hardware are you using for local inference?

You don't need a GPU for fast studio grade voice cloning anymore. Qwen3 TTS (1.7B Q4_K_M) + mainline llama.cpp is officially the fastest way to generate zero shot voice clones using 100% pure CPU execution. Following up on my last post where we ran the Q8 model on a GPU, we just took local C++ voice synthesis a massive step further. The open source community quantized Alibaba's SOTA Qwen3 TTS model down to Q4_K_M GGUF, completely freeing local audio pipelines from dedicated graphics hardware. Here is the real world benchmark and hardware breakdown of running SOTA voice cloning on CPU: # Architecture & Model Setup Using Qwen3-TTS-12Hz-1.7B-Base-Q4_K_M.gguf paired with the 8 bit multimodal projector (mmproj-Q8_0.gguf), llama.cpp executes the entire pipeline in pure C++. No PyTorch, no CUDA dependencies, and no VRAM bottlenecks. # Real-World Memory Footprint - Baseline RAM: 1.6 GB system idle. - Peak Generation RAM: 8 GB RAM during active voice synthesis. - Requirement: Any basic machine with at least 8 GB of system RAM can run this easily. # Real World CPU Benchmarks - Google Colab Free Tier (Throttled 2 Core CPU): Synthesizes a 5 sec studio quality audio clip (~8 words) in 45 seconds. - Modern Consumer CPU (Intel i5/i7 13th/14th Gen or AMD Ryzen 7000/9000): generation should drop to 5 to 20 seconds (nearly 1:1 real-time generation speed!). # Zero Shot Voice Cloning Quality Pass any 5 to 20 second .wav audio sample to the C++ engine using the --tts-speaker-file flag. It yields clean, natural sounding cloned speech with virtually zero quality loss compared to unquantized FP16 weights. To make testing seamless, I built an updated zero config Google Colab notebook. It pulls the official pre built llama.cpp CPU binaries (zero compilation time!) launches a live Gradio web app right in your browser. Record a 5 second clip from your mic (or drop a .mp3, .wav file), type text, and generate cloned audio on CPU. Native C++ audio models are making edge based, offline AI voice agents a reality. Links to the free Q4 CPU Colab notebook and the Q4_K_M GGUF HuggingFace repository are in the replies below! Which models have you been running on your CPUs? What CPU hardware are you using for local inference?

Alok

57,362 Aufrufe • vor 6 Tagen

Right now, you may not have access to models like GPT‑5.6 Sol, GPT‑4.6 Terra, GPT‑5.6 Luna, Claude Mythos 5, or Claude Fable 5. But you can run something surprisingly powerful today, locally, and completely free. in the next 10 mins on your 8 GB VRAM gaming laptop. Gemma 4 26B A4B QAT (MoE) delivers strong performance on a standard 8 GB VRAM GPU using Ollama, with no API, no usage limits, and no external dependencies. Out of the box, it reaches around 20 tokens per second without any optimizations. Only one command in your terminal: Ollama run gemma4:26b This means: Full offline capability (privacy by default) Zero recurring cost Competitive performance for many real world tasks Fast enough for interactive use on cheap consumer hardware If you're waiting for cutting edge cloud models, you're missing what is already practical today: a capable, local LLM that runs entirely on your own machine.

Right now, you may not have access to models like GPT‑5.6 Sol, GPT‑4.6 Terra, GPT‑5.6 Luna, Claude Mythos 5, or Claude Fable 5. But you can run something surprisingly powerful today, locally, and completely free. in the next 10 mins on your 8 GB VRAM gaming laptop. Gemma 4 26B A4B QAT (MoE) delivers strong performance on a standard 8 GB VRAM GPU using Ollama, with no API, no usage limits, and no external dependencies. Out of the box, it reaches around 20 tokens per second without any optimizations. Only one command in your terminal: Ollama run gemma4:26b This means: Full offline capability (privacy by default) Zero recurring cost Competitive performance for many real world tasks Fast enough for interactive use on cheap consumer hardware If you're waiting for cutting edge cloud models, you're missing what is already practical today: a capable, local LLM that runs entirely on your own machine.

Alok

65,387 Aufrufe • vor 1 Monat

You know that saying "a picture speaks a thousand words"? Now it... can! In addition to being able to upload images to use in your Flow videos, you can also add speech to the images you upload. Get your pet ready for its close-up and a monologue 🎬 Audio generation on Flow is still experimental, but we're excited for you try it out. Add your videos below ⬇️

You know that saying "a picture speaks a thousand words"? Now it... can! In addition to being able to upload images to use in your Flow videos, you can also add speech to the images you upload. Get your pet ready for its close-up and a monologue 🎬 Audio generation on Flow is still experimental, but we're excited for you try it out. Add your videos below ⬇️

Google Labs

47,734 Aufrufe • vor 1 Jahr

1/3 🤖 Meet AgentOS: A Token-efficient, Microkernel AI agent with on-device model routing across CLI, Web UI, and chat. A local router reads every message on your device and sends it to the cheapest model that can still do the job well. You stop overpaying for AI. ⚡ What makes it stand out: ■ Smart on-device model routing across 20+ providers ( bankrbot LLM Gateway, OpenRouter, OpenAI, Anthropic, Ollama, and more). ■ Persistent local memory that survives restarts. ■ A layered security sandbox (Standard, Strict, Locked). ■ 37 built-in skills plus MCP, loaded only when a task needs them, and more ■ One unified gateway for CLI, Web UI, Slack, Telegram, Discord, and more Remember this: AgentOS has been integrated with the Bankr LLM Gateway since day one. That means any Bankr user with Bankr API key can start using AgentOS in minutes. Let the router cook.

1/3 🤖 Meet AgentOS: A Token-efficient, Microkernel AI agent with on-device model routing across CLI, Web UI, and chat. A local router reads every message on your device and sends it to the cheapest model that can still do the job well. You stop overpaying for AI. ⚡ What makes it stand out: ■ Smart on-device model routing across 20+ providers ( bankrbot LLM Gateway, OpenRouter, OpenAI, Anthropic, Ollama, and more). ■ Persistent local memory that survives restarts. ■ A layered security sandbox (Standard, Strict, Locked). ■ 37 built-in skills plus MCP, loaded only when a task needs them, and more ■ One unified gateway for CLI, Web UI, Slack, Telegram, Discord, and more Remember this: AgentOS has been integrated with the Bankr LLM Gateway since day one. That means any Bankr user with Bankr API key can start using AgentOS in minutes. Let the router cook.

AgentOS

198,296 Aufrufe • vor 1 Monat

HOLY MOLY running a 35B model locally on a MacBook shouldn’t be THIS FAST 🤯 Spent my weekend in atomic.chat testing Qwen 35B vs. Qwen 27B on my local machine. I had them generate a fully animated HTML/Canvas car mini-game (demo below), ... and both models breezed through the physics and parallax scrolling without a hitch! The secret sauce here is the Atomic Chat app. Because it's perfectly optimized for Mac and uses Google's new TurboQuant under the hood, you can run heavy open-source models flawlessly while keeping top-tier output quality 👊 Other perks: → ZERO setup required → Access 1,000+ models completely free → 100% offline and private → Zero API limits ... and MUCH more! I dropped the prompt I used in the 🧵↓ Spin it up locally and let me know what you get!

HOLY MOLY running a 35B model locally on a MacBook shouldn’t be THIS FAST 🤯 Spent my weekend in atomic.chat testing Qwen 35B vs. Qwen 27B on my local machine. I had them generate a fully animated HTML/Canvas car mini-game (demo below), ... and both models breezed through the physics and parallax scrolling without a hitch! The secret sauce here is the Atomic Chat app. Because it's perfectly optimized for Mac and uses Google's new TurboQuant under the hood, you can run heavy open-source models flawlessly while keeping top-tier output quality 👊 Other perks: → ZERO setup required → Access 1,000+ models completely free → 100% offline and private → Zero API limits ... and MUCH more! I dropped the prompt I used in the 🧵↓ Spin it up locally and let me know what you get!

Charly Wargnier

100,041 Aufrufe • vor 3 Monaten

One of the best formats for promoting your app for organic and ads👇 A talking person in the corner + a screen recording of your app It works because it feels personal while showing the product in action, which helps drive conversions Best part? It works for any app and you can either record yourself or use an AI avatar like I did here with Arcads

César Álvarez

19,101 Aufrufe • vor 1 Monat

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!

Akshay 🚀

126,122 Aufrufe • vor 9 Monaten

🎬 $PALM Presents: the long-awaited Creator Studio. A token holdings based usage system to use the latest generative AI for animations and graphics. Forget about downloadable tools or paying high subscription fees for no usage for mediocre results. Creator Studio allows you to access the tools our developers use for high-quality animated video making with the latest Generative AI tools. Creator Studio is part of the Parrot Framework that assigns tasks to AI agents when possible. You can navigate in a seamless Web UI, making the quality, three-dimensional, non-trippy AI videos and sequences you've ever wanted. We support up to 100 images per user stored in the cloud for you to make video sequences of up to 5 minutes with. Most importantly, we don't charge you a fee - your usage depends on your amount of $PALM tokens! The usage is reset monthly, so if you suddenly run out and need more points to complete your creation, you know what to do - buy some $PALM. This and more is being deployed live at where you can see a preview.

🎬 $PALM Presents: the long-awaited Creator Studio. A token holdings based usage system to use the latest generative AI for animations and graphics. Forget about downloadable tools or paying high subscription fees for no usage for mediocre results. Creator Studio allows you to access the tools our developers use for high-quality animated video making with the latest Generative AI tools. Creator Studio is part of the Parrot Framework that assigns tasks to AI agents when possible. You can navigate in a seamless Web UI, making the quality, three-dimensional, non-trippy AI videos and sequences you've ever wanted. We support up to 100 images per user stored in the cloud for you to make video sequences of up to 5 minutes with. Most importantly, we don't charge you a fee - your usage depends on your amount of $PALM tokens! The usage is reset monthly, so if you suddenly run out and need more points to complete your creation, you know what to do - buy some $PALM. This and more is being deployed live at where you can see a preview.

PaLM AI - $PALM

14,142 Aufrufe • vor 1 Jahr

Introducing Replit ModelFarm, the fastest and safest way to build your next Generative AI app. Available for free on Hacker and Pro plans till October 15th. It requires zero setup, zero configuration, and zero API keys. With Replit ModelFarm, you can build a working Gen AI app in as little as 3 lines of code. Get started by installing the Replit AI library in any Python, JavaScript, or TypeScript Repl. The library implements an API for text completion, chat completion, and text embeddings. It supports streaming so your users can see model responses in real-time rather than waiting on a single output. All Hacker and Pro builders will have free access to a selection of Gen AI models offered by Google Cloud Vertex AI through Replit ModelFarm. All models are accessible from the development environment and any deployed app.

Introducing Replit ModelFarm, the fastest and safest way to build your next Generative AI app. Available for free on Hacker and Pro plans till October 15th. It requires zero setup, zero configuration, and zero API keys. With Replit ModelFarm, you can build a working Gen AI app in as little as 3 lines of code. Get started by installing the Replit AI library in any Python, JavaScript, or TypeScript Repl. The library implements an API for text completion, chat completion, and text embeddings. It supports streaming so your users can see model responses in real-time rather than waiting on a single output. All Hacker and Pro builders will have free access to a selection of Gen AI models offered by Google Cloud Vertex AI through Replit ModelFarm. All models are accessible from the development environment and any deployed app.

Replit ⠕

229,285 Aufrufe • vor 2 Jahren

Hi everyone, my name is Aderinola Agbaje Shakirah, and I own Shaire Laundry Detergent Sheets. Shaire is a modern, eco-friendly detergent sheet that’s powerful on stains, gentle on fabrics, dissolves completely in water, and is perfect for both hand and machine washing. It’s lightweight, travel-friendly, mess-free, and eliminates the need for bulky detergent containers. I’m using this opportunity to ask for your support. 🙏🏾 If you ever come across Shaire Laundry Detergent Sheets in any supermarket or store, please give it a try and patronize my brand. Every purchase means more than you can imagine. Thank you so much for your support. ❤️

Derin.

56,676 Aufrufe • vor 1 Monat

Today we’re open-sourcing Stable Audio Open Small, a 341M-parameter text-to-audio model optimized to run entirely on Arm CPUs. This means 99% of smartphones can now generate music-production samples in seconds, right on-device with no internet required. Built for fast, on-the-go creation, it turns your next quick idea into up to 11 seconds of audio. Generate drum loops, foley, riffs, and textures right where you are. No cords 🔌 just chords 🎹 You can learn more here:

Today we’re open-sourcing Stable Audio Open Small, a 341M-parameter text-to-audio model optimized to run entirely on Arm CPUs. This means 99% of smartphones can now generate music-production samples in seconds, right on-device with no internet required. Built for fast, on-the-go creation, it turns your next quick idea into up to 11 seconds of audio. Generate drum loops, foley, riffs, and textures right where you are. No cords 🔌 just chords 🎹 You can learn more here:

Stability AI

94,796 Aufrufe • vor 1 Jahr

A few things our YouTube teams launched this week: 🕹️ YouTubecreators can now play select games (Playables) directly from their mobile vertical livestreams, easily connecting with their audiences at any time, any place. 🎁 We're rolling out Gifts to horizontal livestreams on mobile, so that creators get another fun, interactive way to earn. 🎶 You can now access the entire Shorts Audio library in our YouTube Create app, which has a suite of editing tools for easily creating videos and uploading them to YouTube.

A few things our YouTube teams launched this week: 🕹️ YouTubecreators can now play select games (Playables) directly from their mobile vertical livestreams, easily connecting with their audiences at any time, any place. 🎁 We're rolling out Gifts to horizontal livestreams on mobile, so that creators get another fun, interactive way to earn. 🎶 You can now access the entire Shorts Audio library in our YouTube Create app, which has a suite of editing tools for easily creating videos and uploading them to YouTube.

Neal Mohan

132,643 Aufrufe • vor 3 Monaten

I just open-sourced my /learn skill. Learn anything with agents and HTML artifacts. I have been learning about all kinds of topics with it. Install the skill and interact with any agent to help you through any topic. Ask it to generate visual and interactive artifacts and help you go deeper or generate knowledge checks (e.g., quizzes). Upskilling myself on any topic is one of the most impactful ways I have been able to use AI agents. If you are a DAIR Academy pro member, you can use it with our AI Builder. Skill: Try now:

I just open-sourced my /learn skill. Learn anything with agents and HTML artifacts. I have been learning about all kinds of topics with it. Install the skill and interact with any agent to help you through any topic. Ask it to generate visual and interactive artifacts and help you go deeper or generate knowledge checks (e.g., quizzes). Upskilling myself on any topic is one of the most impactful ways I have been able to use AI agents. If you are a DAIR Academy pro member, you can use it with our AI Builder. Skill: Try now:

elvis

34,407 Aufrufe • vor 2 Monaten

This tool is literally Higgsfield AI but FREE for good. It's called Wan2GP. A full AI video studio built specifically for people without expensive hardware. Runs on as little as 6GB of VRAM, even old RTX 10-series cards and 8GB laptops. Everything stays on your machine, no uploads, no caps, no watermarks. What you get in one app: • Text-to-video and image-to-video generation • The best open models built in: Wan 2.2, LTX-2, Hunyuan Video, Flux • A full browser interface with a queue system • LoRA support to customize any model • Mask editor and prompt enhancer included A 5-second clip generates in minutes on a mid-range gaming rig. No subscription, ever. 100% Free. Open Source.

This tool is literally Higgsfield AI but FREE for good. It's called Wan2GP. A full AI video studio built specifically for people without expensive hardware. Runs on as little as 6GB of VRAM, even old RTX 10-series cards and 8GB laptops. Everything stays on your machine, no uploads, no caps, no watermarks. What you get in one app: • Text-to-video and image-to-video generation • The best open models built in: Wan 2.2, LTX-2, Hunyuan Video, Flux • A full browser interface with a queue system • LoRA support to customize any model • Mask editor and prompt enhancer included A 5-second clip generates in minutes on a mid-range gaming rig. No subscription, ever. 100% Free. Open Source.

Simplifying AI

175,970 Aufrufe • vor 16 Tagen

The official unveiling of the brand new Specs Glasses from Snapchat $2195 Pre-order today at Release date : Fall 2026 4 hours of mixed usage battery life Up to 20 hours battery life with the case Apps and Games Cameras Built in speakers These glasses are completely hands-free and do not need to be connected to any devices. If you want to connect them to devices you can connect them to a computer or a phone or tablet and view the screen through the glasses. #specs #smartglasses #tech powered by Snapdragon #xrglasses

GregglesTV

30,811 Aufrufe • vor 1 Monat

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵

A major question in multimodal modeling is how to leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵

Amir Zamir

23,628 Aufrufe • vor 1 Monat