Throughout my journey in developing multimodal models, I’ve always... wanted a framework that lets me plug & play modality encoders/decoders on top of an auto-regressive LLM. I want to prototype fast, try new architectures, and have my demo files scale effortlessly — with full support for parallelism and optimization. Not just to hack⚙️, but also to scale🚀. So finally we built it for ourselves. LMMs-Engine: a lean, efficient framework built to train unified multimodal model at scale. From Qwen LLM, VLM, LLaVA-OV, and WanVideo, to unified models like Qwen-Omni and BAGEL — plus Linear-Attn GDN and research prototypes like RAE and SiT - all under one modular system that seamlessly integrates diverse datasets and optimization strategies. Powered by FSDP2 multi-dim parallelism, Ulysses sequence parallel, Flash-Attention, Liger Kernels, and Native Sparse Attention (also with bonus support for the Muon optimizer for all models).show more

A major question in multimodal modeling is how to... leverage strong pre-trained models, such as industry-scale LLMs and VLMs, during training to avoid starting from scratch. This is important as the demand for rich and domain-specific multimodal models continues to increase, while training them from scratch is obviously impractical due to limited data and compute. The so-called “any-to-any” multimodal models can model a large diverse dictionary of modalities. That’s good. Their downside is that their architectures are often not decoder-only, which has limited their performance in practice and prevents them from leveraging strong pre-trained decoder-only models as priors. We are releasing MODUS (ICML Conference '26) a decoder-only any-to-any multimodal model to address some of these questions. A single transformer decoder predicts any modality from any others with no modality-specific heads, losses, or task pipelines. We show efficient adaptation of established models, e.g., BAGEL, to rich any-to-any multimodal modeling. We are releasing 14B to 77B parameter models. All materials are open-source. The download links and demos here 🧵show more

🚨 New: Integrating Harbor (Harbor Framework) for end-to-end Computer-Use... evaluation(for Windows and Linux) at scale with Thinking Machines' Tinker, OSWorld, Daytona, and bare-metal servers. We just added support for Computer Use, Tinker, and OSWorld to Harbor - a framework for evaluating agents and generating RL training data by running large-scale rollouts across parallel sandboxed environments and collecting trajectories for SFT and RL. Repo and blogpost below 👇show more

GROK DOMINATES AI AT SCALE, REDEFINING WHAT’S POSSIBLE IN... REAL TIME Most AI models stall at the prototype phase. Grok advances beyond limits where others fail to perform. •⁠ ⁠Built for real-time decisioning across vast data landscapes •⁠ ⁠Optimized to process, synthesize, and respond with human-like fluidity •⁠ ⁠Deploys intelligence at the edge, not just in the cloud •⁠ ⁠Powers robotics, research, and infrastructure in live environments •⁠ ⁠Infrastructure built to evolve, not collapse, under scale Grok delivers high-performance AI built for deployment and real-world use. Source: xAI, Grok, Elon Muskshow more

Show-o One Single Transformer to Unify Multimodal Understanding and... Generation discuss: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model.show more

NVIDIA announces new Isaac tools for agents as Agile... Robots unveils Agile ONE SN, a reference humanoid for robotics research. 🦾 Built on NVIDIA Isaac GR00T and powered by Jetson Thor, Agile ONE SN gives researchers and developers an integrated platform to develop, train and validate humanoid skills. The Isaac expansion also introduces new datasets, a runtime, and robot learning framework updates to accelerate development from data to deployment. Read the #NVIDIAGTC announcement:show more

We are entering an extremely exciting era for open-weight... models. Kimi K2.6 now feels like a top agentic model. I took it for a spin via Fireworks AI fast inference APIs. Kimi K2.6 has impressive agentic capabilities, design skills, and the ability to synthesize large amounts of information. I built a little Skill that produces survey papers on any AI research topic you want. (see example in the clip) You can use the skill to tell your agent to generate a survey on whatever topic and watch it go to work. The artifact was fully generated by Kimi.ai's Kimi K2.6. It's cheap and fast. Next step for me is to explore ways to continue integrating the capabilities of these models on use cases like automating my LLM knowledge bases and augmenting my agent memory capabilities. Stay tuned for more.show more

Generalist robots need multimodal learning—fusing vision, touch, and body... motion to handle unstructured worlds. 🦾 NVIDIA Isaac Lab is a GPU-accelerated, open-source simulation framework for robot learning that unifies physics, rendering, sensing, and parallel computing to train robot policies at scale. Learn more 🔗 #NVIDIAResearchshow more

Massive update for AI Engineers! Training diffusion models just... got a lot easier. dLLM is an open-source library that does for diffusion models what Hugging Face did for transformers. Here's why this matters: Traditional autoregressive models generate text left-to-right, one token at a time. Diffusion models work differently - they refine the entire sequence iteratively, giving you better control over generation quality and more flexible editing capabilities. The problem? Building and training these models required stitching together scattered tools and reimplementing research papers from scratch. Most of the tooling has been scattered and is hard to reproduce. dLLM changes this: It unifies everything you need to train, evaluate, and deploy diffusion language models: ↳ Scalable training with LoRA, DeepSpeed, and FSDP support ↳ Unified evaluation that abstracts away inference complexity ↳ Ready-to-use recipes for pretraining, finetuning, and evaluation The library includes implementations of models like LLaDA and Dream, plus training algorithms like Edit Flows that enable insertion, deletion, and substitution operations. The team just released ModernBERT-Chat models showing you can turn BERT into lightweight chatbots through masked instruction tuning. This is practical and worth exploring. The setup is straightforward. Run locally with Accelerate or scale to multi-node clusters with Slurm. If you're working with language models and want to explore diffusion-based approaches without rebuilding infrastructure, dLLM gives you a production-ready starting point. Link to the repo in the next tweet.show more

We just hit a new huge milestone. 🏆 Thanks... to our amazing community, we have now reached 170,000 3D Maps across the world. Here are the numbers that define what we’ve built so far: ↳ 170,000 mapped locations ↳ 767 TB of world data ↳ 85.2M+ images captured ↳ And a steady, unstoppable month over month growth. A single dataset designed to help machines see, understand, and navigate space, also used to train spatial AI and visual models, and to geolocalize robots and experiences with precision through our VPS, in order to interact seamlessly with the real world. And yes... you read that right! We are already in talks with several top AI companies that want access to our dataset to train their AI models and to support what comes next, including robotics and advanced visual and video models. It is all one story, one ecosystem, one vision, and it is called Over the Reality.show more

This is native Qwen Image 2.1, no LoRA -... move and resize objects seamlessly in an image🔥 I was planning to finetune on this task to showcase the new 🧨 diffusers LoRA support for Qwen 2.1 (go try it!), but while testing the base model I realized it's actually so good at it out of the box (pun intended haha) Try it ▶️ Train 🧨show more

AI-first aircraft development: parametric geometry generated at scale without... failures, Physics AI models reporting lift and drag across a wide design space, and multi-objective optimization to get to the most performant aircraft. All in minutes instead of weeks or months.show more

I wanted to test one of the models out... and so I made a chica walking cycle for the fun of it. ============= Please note that it's not suppose to be really accurate to the movie, but have that taste from the movie. THIS IS ALSO NOT RELATED TO MY PROJECT!!show more

We are thrilled to announce that we have the... support Solana as we build and scale the next $STAGE of our journey. Together, we’re paving the way for innovation and growth 🚀show more

Today we're officially launching Moby 2 to the world... (and to our full house at The Whalies in Los Angeles). It is the only AI ecommerce brands can trust to do work for them. We’re also launching: • Moby Specialists, purpose-built AI specialists for media buying, creative, and conversion optimization • Compass, our new unified measurement system combining attribution, incrementality, and MMM • New upgrades to Sonar Optimize (including attribution passback) and Clicks & Deterministic Views Attribution Built on data from 60,000+ brands and $82B in tracked revenue, Triple Whale is the AI operating system for ecommerce, and Moby 2 is the operator, always working for you. See it in action:show more

Dexter can now use local LLMs. We finally added... ollama support today. Includes models like: • qwen 3 • gemma 3 • deepseek r1 Because Dexter is built on LangChain, switching between local and cloud models is easy. Huge thanks to Harsh Gupta for shipping this.show more

At Applied Compute, we're not just training models, we're... building a model factory. To help researchers scale, we built Ari, our in-house AI research agent. Ari diligently sifts through gigabytes of logs to find evidence of unhealthy behavior, remembers results the team has learned from past experiments, and produces styled research reports for the team to review.show more

Introducing Jan v3, with updates to Jan Desktop v0.7.6... 💛 Jan v3 is our first v3 model, a 4B base built for fine-tuning and fast local use, with stronger math and coding. This release also includes a small Jan Desktop update, starting with a UI refresh as we move toward a more unified Jan experience. Try it: - Jan v3 is available in Jan Desktop and at - Get the latest desktop app at Model: - Jan-v3-4B: - Jan-v3-4B-GGUF: Thanks Qwen for the base model and Georgi Gerganov for llama.cpp 💛show more

Introducing Arcana. The only framework that lets you build... and run a complete onchain app in one place. Built for the way people build now. Prompt it, deploy it, run it onchain, all from one framework with no infrastructure to assemble.show more

I built a Claude agent for Meta ads ...that... analyzes your account and tells you exactly what to fix to scale profitably Comment "SCALE" and I'll send you the full workflow 👀show more