Throughout my journey in developing multimodal models, I’ve always... wanted a framework that lets me plug & play modality encoders/decoders on top of an auto-regressive LLM. I want to prototype fast, try new architectures, and have my demo files scale effortlessly — with full support for parallelism and optimization. Not just to hack⚙️, but also to scale🚀. So finally we built it for ourselves. LMMs-Engine: a lean, efficient framework built to train unified multimodal model at scale. From Qwen LLM, VLM, LLaVA-OV, and WanVideo, to unified models like Qwen-Omni and BAGEL — plus Linear-Attn GDN and research prototypes like RAE and SiT - all under one modular system that seamlessly integrates diverse datasets and optimization strategies. Powered by FSDP2 multi-dim parallelism, Ulysses sequence parallel, Flash-Attention, Liger Kernels, and Native Sparse Attention (also with bonus support for the Muon optimizer for all models).show more

🚨 New: Integrating Harbor (Harbor Framework) for end-to-end Computer-Use... evaluation(for Windows and Linux) at scale with Thinking Machines' Tinker, OSWorld, Daytona, and bare-metal servers. We just added support for Computer Use, Tinker, and OSWorld to Harbor - a framework for evaluating agents and generating RL training data by running large-scale rollouts across parallel sandboxed environments and collecting trajectories for SFT and RL. Repo and blogpost below 👇show more

GROK DOMINATES AI AT SCALE, REDEFINING WHAT’S POSSIBLE IN... REAL TIME Most AI models stall at the prototype phase. Grok advances beyond limits where others fail to perform. •⁠ ⁠Built for real-time decisioning across vast data landscapes •⁠ ⁠Optimized to process, synthesize, and respond with human-like fluidity •⁠ ⁠Deploys intelligence at the edge, not just in the cloud •⁠ ⁠Powers robotics, research, and infrastructure in live environments •⁠ ⁠Infrastructure built to evolve, not collapse, under scale Grok delivers high-performance AI built for deployment and real-world use. Source: xAI, Grok, Elon Muskshow more

Show-o One Single Transformer to Unify Multimodal Understanding and... Generation discuss: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model.show more

Something NVIDIA & Google do better than anyone else... is software-hardware-system co-design, and not just optimizing hardware for current model architectures, but predicting future ones. Back in early 2022, when NVIDIA started the design process for NVL72, MoE (Mixture of Experts) models were not yet the standard, and dense models were still dominant for frontier models. However, NVIDIA's strong software-hardware co-design culture enabled them to make a calculated bet that MoEs were the future, and they built NVL72 specifically for best MoE performance per TCO (Total Cost of Ownership). Furthermore, back in 2022, disaggregated prefill and wide expert parallelism (wideEP) MoE inference optimizations hadn't been invented yet, but it turns out that these MoE inference optimizations work best on large-scale systems like NVL72. While most other AI chip companies' in-house AI labs focus on training small 5B models that mainly use data parallelism, NVIDIA and Google's in-house AI labs continuously push the boundaries of model architecture and training recipes, such as NVFP4 training. Just like Super Idol & IShowSpeed, there must be a strong partnership between software engineers and hardware engineers to deliver the best systems that maximize performance per TCO.show more

We are entering an extremely exciting era for open-weight... models. Kimi K2.6 now feels like a top agentic model. I took it for a spin via Fireworks AI fast inference APIs. Kimi K2.6 has impressive agentic capabilities, design skills, and the ability to synthesize large amounts of information. I built a little Skill that produces survey papers on any AI research topic you want. (see example in the clip) You can use the skill to tell your agent to generate a survey on whatever topic and watch it go to work. The artifact was fully generated by Kimi.ai's Kimi K2.6. It's cheap and fast. Next step for me is to explore ways to continue integrating the capabilities of these models on use cases like automating my LLM knowledge bases and augmenting my agent memory capabilities. Stay tuned for more.show more

Generalist robots need multimodal learning—fusing vision, touch, and body... motion to handle unstructured worlds. 🦾 NVIDIA Isaac Lab is a GPU-accelerated, open-source simulation framework for robot learning that unifies physics, rendering, sensing, and parallel computing to train robot policies at scale. Learn more 🔗 #NVIDIAResearchshow more

I wanted to test one of the models out... and so I made a chica walking cycle for the fun of it. ============= Please note that it's not suppose to be really accurate to the movie, but have that taste from the movie. THIS IS ALSO NOT RELATED TO MY PROJECT!!show more

Dexter can now use local LLMs. We finally added... ollama support today. Includes models like: • qwen 3 • gemma 3 • deepseek r1 Because Dexter is built on LangChain, switching between local and cloud models is easy. Huge thanks to Harsh Gupta for shipping this.show more

We are thrilled to announce that we have the... support Solana as we build and scale the next $STAGE of our journey. Together, we’re paving the way for innovation and growth 🚀show more

Introducing Jan v3, with updates to Jan Desktop v0.7.6... 💛 Jan v3 is our first v3 model, a 4B base built for fine-tuning and fast local use, with stronger math and coding. This release also includes a small Jan Desktop update, starting with a UI refresh as we move toward a more unified Jan experience. Try it: - Jan v3 is available in Jan Desktop and at - Get the latest desktop app at Model: - Jan-v3-4B: - Jan-v3-4B-GGUF: Thanks Qwen for the base model and Georgi Gerganov for llama.cpp 💛show more

Introducing Arcana. The only framework that lets you build... and run a complete onchain app in one place. Built for the way people build now. Prompt it, deploy it, run it onchain, all from one framework with no infrastructure to assemble.show more

Today we're officially launching Moby 2 to the world... (and to our full house at The Whalies in Los Angeles). It is the only AI ecommerce brands can trust to do work for them. We’re also launching: • Moby Specialists, purpose-built AI specialists for media buying, creative, and conversion optimization • Compass, our new unified measurement system combining attribution, incrementality, and MMM • New upgrades to Sonar Optimize (including attribution passback) and Clicks & Deterministic Views Attribution Built on data from 60,000+ brands and $82B in tracked revenue, Triple Whale is the AI operating system for ecommerce, and Moby 2 is the operator, always working for you. See it in action:show more

Happy to introduce my video on this alternative LLM... architecture, Mamba and State Space Models! I wanted to do it for a while now and finally found the time to work on animating my visual guide. Expect many, many, many visuals! Link in comment 👇show more

I built a Claude agent for Meta ads ...that... analyzes your account and tells you exactly what to fix to scale profitably Comment "SCALE" and I'll send you the full workflow 👀show more

We are excited to launch our two models Pharia-1-LLM-7B-control... and Pharia-1-LLM-7B-control-aligned. Both models and the code used to train them are now publicly available and open-sourced for non-commercial research and educational use. Read our model blog post here: Learn more about our open-source codebase Scaling: #writtenbyalephalphashow more

We’re not just training models anymore. We’re deploying intelligence... at scale. On the latest episode of Advanced Insights, AMD CTO MarkPapermaster sits down with Oracle's Chris Gandolfo to talk AI, infrastructure, and what it really takes to train and deploy LLMs at scale.show more

Backlog → shipped. Engine is a remote AI software... engineer built for pros. It integrates with all of your existing tools and supports any frontier LLM.show more

1/ Happy to share UniDisc - Unified Multimodal Discrete... Diffusion – We train a 1.5 billion parameter transformer model from scratch on 250 million image/caption pairs using a **discrete diffusion objective**. Our model has all the benefits of diffusion models but now in multimodal space! - flexible compute-quality tradeoff, zero-shot inpainting and editing, better control via classifier-free guidance and lower latency! We open source everything - our code, weights and the training dataset.show more

Social media really messes with your head and your... sense of reality, especially when it comes to what you’re supposed to look like. Women’s bodies come in all shapes and sizes! I’ve birthed 3 babies! I have extra skin, I’ll always have a “belly” (I’ve had a curve to my stomach since HS no matter how tiny I was) some women are built like that. I have hip dips and cellulite. And guess what I love my body! I workout and fuel my body with healthy foods to be strong and to enjoy my life and not be a slave to a scale or mirror. Let’s normalize normal bodies! ALL BODIES ARE BEAUTIFUL💗 #bodypositivityshow more

NVIDIA AI Released DiffusionRenderer: An AI Model for Editable,... Photorealistic 3D Scenes from a Single Video In a groundbreaking new paper, researchers at NVIDIA, University of Toronto, Vector Institute and the University of Illinois Urbana-Champaign have unveiled a framework that directly tackles this challenge. DiffusionRenderer represents a revolutionary leap forward, moving beyond mere generation to offer a unified solution for understanding and manipulating 3D scenes from a single video. It effectively bridges the gap between generation and editing, unlocking the true creative potential of AI-driven content. DiffusionRenderer treats the “what” (the scene’s properties) and the “how” (the rendering) in one unified framework built on the same powerful video diffusion architecture that underpins models like Stable Video Diffusion..... Read full article here: Paper: GitHub Page: NVIDIA NVIDIA AI NVIDIAnewsroom NVIDIA AIDevshow more