Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

VideoLLaMA3, latest MLLMs for image and video understanding. 🖐️ 7B models: DocVQA: 94.9, MathVision: 26.2, VideoMME: 66.2/70.3, MLVU: 73.0 🤏 2B models for edge devices: MMMU: 45.3, VideoMME: 59.6/63.4 👊 Frontier-class video model with ONLY 3M video-text pairs

Gradio

56,759 subscribers

37,154 views • 1 year ago •via X (Twitter)

Science & Technology

Anya Rossi• Live Now

Private livecam show

7 Comments

Gradio1 year ago

VideoLLaMA3 for Video understanding:

Gradio1 year ago

VideoLLaMA3 for Image understanding:

Gradio1 year ago

Learn more about VideoLLaMA3:

Rainmaker2 years ago

Join me as I put several Machine Learning models head-to-head to see which one can beat the market and deliver strong returns. In this free Substack post I share several models that deliver better returns with much lower drawdown compared to Buy-and-Hold approach.

Discord Guys1 year ago

Wow, VideoLLaMA3 sounds like a beast! Can't wait to try it out.

Sujith Reddy1 year ago

@_akhaliq @Gradio are these models production level models? Scalable?

ヌル1 year ago

3M pairs is a small dataset for training.

Related Videos

🤯DiffIR2VR-Zero: Zero-shot video restoration to high-resolution using pre-trained image restoration diffusion models. - Framework can handle video denoising and up to 8x super-resolution - outperforms trained models in generalizing across diverse datasets and extreme degradations - Compatible with every 2D restoration model

🤯DiffIR2VR-Zero: Zero-shot video restoration to high-resolution using pre-trained image restoration diffusion models. - Framework can handle video denoising and up to 8x super-resolution - outperforms trained models in generalizing across diverse datasets and extreme degradations - Compatible with every 2D restoration model

Gradio

20,354 views • 2 years ago

Meet Reka Edge – Our next-generation vision language model for physical AI. Uses 3x fewer input tokens and achieves 65% faster throughput compared to leading 8B models. Image understanding, video analysis, object detection, and tool use. Built for Action. Fast enough for production, deployable anywhere. Read more:

Meet Reka Edge – Our next-generation vision language model for physical AI. Uses 3x fewer input tokens and achieves 65% faster throughput compared to leading 8B models. Image understanding, video analysis, object detection, and tool use. Built for Action. Fast enough for production, deployable anywhere. Read more:

Reka

54,012 views • 5 months ago

🤯 OneDiffusion: A versatile, large-scale diffusion model that seamlessly supports bidirectional image synthesis and understanding across diverse tasks. ✅ Text to Image ✅ Image to Depth ✅ Image to Segmentation ✅ Image to Pose ✅ FaceID ✅ Image to Multiview How to use & more👇

🤯 OneDiffusion: A versatile, large-scale diffusion model that seamlessly supports bidirectional image synthesis and understanding across diverse tasks. ✅ Text to Image ✅ Image to Depth ✅ Image to Segmentation ✅ Image to Pose ✅ FaceID ✅ Image to Multiview How to use & more👇

Gradio

11,820 views • 1 year ago

MiniCPM-V 4.5 achieves an average score of 77.0 on OpenCompass, a comprehensive evaluation of 8 popular benchmarks. With only 8B parameters, it surpasses widely used proprietary models like GPT-4o-latest, Gemini-2.0 Pro, and strong open-source models like Qwen2.5-VL 72B powered by a new unified 3D-Resampler over images and videos, MiniCPM-V 4.5 can now achieve 96x compression rate for video tokens, where 6 448x448 video frames can be jointly compressed into 64 video tokens (normally 1,536 tokens for most MLLMs). vibe coding a Video Chat AI app with MiniCPM-V-4.5 in anycoder

MiniCPM-V 4.5 achieves an average score of 77.0 on OpenCompass, a comprehensive evaluation of 8 popular benchmarks. With only 8B parameters, it surpasses widely used proprietary models like GPT-4o-latest, Gemini-2.0 Pro, and strong open-source models like Qwen2.5-VL 72B powered by a new unified 3D-Resampler over images and videos, MiniCPM-V 4.5 can now achieve 96x compression rate for video tokens, where 6 448x448 video frames can be jointly compressed into 64 video tokens (normally 1,536 tokens for most MLLMs). vibe coding a Video Chat AI app with MiniCPM-V-4.5 in anycoder

AK

19,088 views • 11 months ago

Video understanding isn't just recognizing —it demands reasoning across thousands of frames. Meet Long-RL🚀 Highlights: 🧠 Dataset: LongVideo-Reason — 52K QAs with reasoning. ⚡ System: MR-SP - 2.1× faster RL for long videos. 📈 Scalability: Hour-long videos (3,600 frames) RL on a single node (8×A100s). 🖼️📝🎵 RL training for video, text, audio — works with VILA, Qwen series, and image/video generation models 🎨🎬 📄 Paper: 🎥 Demo: 💻 Code:

Video understanding isn't just recognizing —it demands reasoning across thousands of frames. Meet Long-RL🚀 Highlights: 🧠 Dataset: LongVideo-Reason — 52K QAs with reasoning. ⚡ System: MR-SP - 2.1× faster RL for long videos. 📈 Scalability: Hour-long videos (3,600 frames) RL on a single node (8×A100s). 🖼️📝🎵 RL training for video, text, audio — works with VILA, Qwen series, and image/video generation models 🎨🎬 📄 Paper: 🎥 Demo: 💻 Code:

Yukang Chen

31,807 views • 1 year ago

SITUATION EXPLAINED: MiniMax released H3 with open weights, and it's already #1 in video editing. • Priced at $7.80 per minute of 2K video, against Seedance 2.0 at $22.45 and Kling 3.0 at $20.16 • It ranks #1 on Artificial Analysis for video editing, #2 for text-to-video behind Gemini Omni Flash, and #3 for image-to-video • H3 takes text, images, video, and audio in a single context and generates 5-15 second clips at native 2K, 24fps, with native stereo audio • MiniMax is releasing the weights under its Community License within days, which would make it the strongest open-weights video model by a wide margin • It launched the same week ByteDance rolled out Seedance 2.5, which now generates 30-second 4K video from a single prompt • Seedance 2.5's face consistency across a full 30 seconds is the visible jump, earlier models drifted into a different face partway through • Voices are still the obvious tell, the video reads as real long before the audio does • Smaller labs building purpose-built video models are now beating frontier labs at this specific task Theo Jaffee: "This is gonna be one of the biggest examples of things that people are gonna do with these models, historical simulations, basically. Nostalgia videos for zoomers."

SITUATION EXPLAINED: MiniMax released H3 with open weights, and it's already #1 in video editing. • Priced at $7.80 per minute of 2K video, against Seedance 2.0 at $22.45 and Kling 3.0 at $20.16 • It ranks #1 on Artificial Analysis for video editing, #2 for text-to-video behind Gemini Omni Flash, and #3 for image-to-video • H3 takes text, images, video, and audio in a single context and generates 5-15 second clips at native 2K, 24fps, with native stereo audio • MiniMax is releasing the weights under its Community License within days, which would make it the strongest open-weights video model by a wide margin • It launched the same week ByteDance rolled out Seedance 2.5, which now generates 30-second 4K video from a single prompt • Seedance 2.5's face consistency across a full 30 seconds is the visible jump, earlier models drifted into a different face partway through • Voices are still the obvious tell, the video reads as real long before the audio does • Smaller labs building purpose-built video models are now beating frontier labs at this specific task Theo Jaffee: "This is gonna be one of the biggest examples of things that people are gonna do with these models, historical simulations, basically. Nostalgia videos for zoomers."

MTS

55,187 views • 27 days ago

Meta announces Movie Gen A Cast of Media Foundation Models We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user’s image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models

Meta announces Movie Gen A Cast of Media Foundation Models We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user’s image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models

AK

62,719 views • 1 year ago

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

AK

143,469 views • 3 years ago

📣 Microsoft Research releases Florence-VL, a new family of MLLMs powered by the generative vision foundation model Florence-2. Achieves significant improvements in general VQA, perception, hallucination, OCR, Chart, knowledge-intensive understanding, and more🔥Learn more👇

📣 Microsoft Research releases Florence-VL, a new family of MLLMs powered by the generative vision foundation model Florence-2. Achieves significant improvements in general VQA, perception, hallucination, OCR, Chart, knowledge-intensive understanding, and more🔥Learn more👇

Gradio

14,454 views • 1 year ago

Object Cutter Create high-quality HD background removal for ANY object in your image with a text prompt or bounding boxes!

Object Cutter Create high-quality HD background removal for ANY object in your image with a text prompt or bounding boxes!

Gradio

136,042 views • 1 year ago

InternLM-XComposer-2.5 A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision language model that supports long-contextual input and output. IXC-2.5 excels in various text-image comprehension and composition applications, achieving GPT-4V level capabilities with merely 7B LLM backend. Trained with 24K interleaved image-text contexts, it can seamlessly extend to 96K long contexts via RoPE extrapolation. This long-context capability allows IXC-2.5 to excel in tasks requiring extensive input and output contexts. Compared to its previous 2.0 version, InternLM-XComposer-2.5 features three major upgrades in vision-language comprehension: (1) Ultra-High Resolution Understanding, (2) Fine-Grained Video Understanding, and (3) Multi-Turn Multi-Image Dialogue. In addition to comprehension, IXC-2.5 extends to two compelling applications using extra LoRA parameters for text-image composition: (1) Crafting Webpages and (2) Composing High-Quality Text-Image Articles. IXC-2.5 has been evaluated on 28 benchmarks, outperforming existing open-source state-of-the-art models on 16 benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on 16 key tasks.

InternLM-XComposer-2.5 A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision language model that supports long-contextual input and output. IXC-2.5 excels in various text-image comprehension and composition applications, achieving GPT-4V level capabilities with merely 7B LLM backend. Trained with 24K interleaved image-text contexts, it can seamlessly extend to 96K long contexts via RoPE extrapolation. This long-context capability allows IXC-2.5 to excel in tasks requiring extensive input and output contexts. Compared to its previous 2.0 version, InternLM-XComposer-2.5 features three major upgrades in vision-language comprehension: (1) Ultra-High Resolution Understanding, (2) Fine-Grained Video Understanding, and (3) Multi-Turn Multi-Image Dialogue. In addition to comprehension, IXC-2.5 extends to two compelling applications using extra LoRA parameters for text-image composition: (1) Crafting Webpages and (2) Composing High-Quality Text-Image Articles. IXC-2.5 has been evaluated on 28 benchmarks, outperforming existing open-source state-of-the-art models on 16 benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on 16 key tasks.

AK

66,472 views • 2 years ago

We’re excited to announce the release and open-source of HunyuanImage 3.0 — the largest and most powerful open-source text-to-image model to date, with over 80 billion total parameters, of which 13 billion are activated per token during inference.The effect is completely comparable to the industry’s flagship closed-source model.🚀🚀🚀 HunyuanImage 3.0 originates from our internally developed native multimodal large language model, with fine-tuning and post-training focused on text-to-image generation. This unique foundation gives the model a powerful set of capabilities: ✅Reason with world knowledge ✅Understand complex, thousand-word prompts ✅Generate precise text within images Different from traditional DiT architecture image generation models, HunyuanImage 3.0’s MoE architecture uses a Transfusion-based approach to deeply couple Diffusion and LLM training for a single, powerful system. Built on Hunyuan-A13B, HunyuanImage 3.0 was trained on a massive dataset: 5 billion image-text pairs, video frames, interleaved image-text data, and 6 trillion tokens of text corpora. This hybrid training across multimodal generation, understanding, and LLM capabilities allows the model to seamlessly integrate multiple tasks. Whether you're an illustrator, designer, or creator, this is built to slash your workflow from hours to minutes. HunyuanImage 3.0 can generate intricate text, detailed comics, expressive emojis, and lively, engaging illustrations for educational content. The current release focuses solely on text-to-image generation and future updates will include image-to-image, image editing, multi-turn interaction, and more. 👉🏻Try it now: 🔗GitHub: 🤗Hugging Face:

We’re excited to announce the release and open-source of HunyuanImage 3.0 — the largest and most powerful open-source text-to-image model to date, with over 80 billion total parameters, of which 13 billion are activated per token during inference.The effect is completely comparable to the industry’s flagship closed-source model.🚀🚀🚀 HunyuanImage 3.0 originates from our internally developed native multimodal large language model, with fine-tuning and post-training focused on text-to-image generation. This unique foundation gives the model a powerful set of capabilities: ✅Reason with world knowledge ✅Understand complex, thousand-word prompts ✅Generate precise text within images Different from traditional DiT architecture image generation models, HunyuanImage 3.0’s MoE architecture uses a Transfusion-based approach to deeply couple Diffusion and LLM training for a single, powerful system. Built on Hunyuan-A13B, HunyuanImage 3.0 was trained on a massive dataset: 5 billion image-text pairs, video frames, interleaved image-text data, and 6 trillion tokens of text corpora. This hybrid training across multimodal generation, understanding, and LLM capabilities allows the model to seamlessly integrate multiple tasks. Whether you're an illustrator, designer, or creator, this is built to slash your workflow from hours to minutes. HunyuanImage 3.0 can generate intricate text, detailed comics, expressive emojis, and lively, engaging illustrations for educational content. The current release focuses solely on text-to-image generation and future updates will include image-to-image, image editing, multi-turn interaction, and more. 👉🏻Try it now: 🔗GitHub: 🤗Hugging Face:

Tencent Hy

412,880 views • 11 months ago

Exciting news – our NetMind Power Inference Platform is officially live! We’ve launched with 12 open-source models in areas such as image generation, text-to-image conversion, and voice synthesis. Users can directly use these models on the Inference Platform or access them via API. The platform is currently in Beta and free to use – so be sure to check it out now before the free Beta period ends! For a detailed demo, follow the video link below. Remember, we’ll continuously add more models, so keep returning for the latest advancements. Explore these powerful AI capabilities today and see what you can create! #NetMindPower #OpenSourceAI #AIInference

Exciting news – our NetMind Power Inference Platform is officially live! We’ve launched with 12 open-source models in areas such as image generation, text-to-image conversion, and voice synthesis. Users can directly use these models on the Inference Platform or access them via API. The platform is currently in Beta and free to use – so be sure to check it out now before the free Beta period ends! For a detailed demo, follow the video link below. Remember, we’ll continuously add more models, so keep returning for the latest advancements. Explore these powerful AI capabilities today and see what you can create! #NetMindPower #OpenSourceAI #AIInference

NetMind.AI

46,274 views • 2 years ago

Explore state-of-the-art multimodal prompting in our new short course Large Multimodal Model Prompting with Gemini, taught by Erwin Huizenga in collaboration with Google Cloud. One interesting insight from this course: with multimodal models, prompt structure matters significantly. Placing text inputs, such as a patient's medical history, before image inputs, like an X-ray, can enhance the model's ability to contextualize and interpret visual data effectively. In other contexts, such as image captioning, you may get better results by putting the image first. Multimodal models behave differently than text-only LLMs, and effective prompting for models varies depending on the model you’re using. In this course you’ll learn how to effectively prompt Gemini models. Gemini's multimodal capabilities also enable new approaches in AI application development, for example: - The Gemini library handles various video formats (MP4, MOV, MPEG), streamlining applications using these formats. - Large context window (up to 1 million tokens) enables processing of extensive content, like analyzing multiple 50-minute videos simultaneously. - Function calling feature integrates real-time data (e.g., current exchange rates) into model responses. The course demonstrates building multimodal applications with real-world examples including document analyzers that reason across text and graphs simultaneously, video content extractors that find and timestamp specific information from multiple hours of footage, and automated expense report systems processing receipt images while cross-referencing company policies. Sign up here:

Explore state-of-the-art multimodal prompting in our new short course Large Multimodal Model Prompting with Gemini, taught by Erwin Huizenga in collaboration with Google Cloud. One interesting insight from this course: with multimodal models, prompt structure matters significantly. Placing text inputs, such as a patient's medical history, before image inputs, like an X-ray, can enhance the model's ability to contextualize and interpret visual data effectively. In other contexts, such as image captioning, you may get better results by putting the image first. Multimodal models behave differently than text-only LLMs, and effective prompting for models varies depending on the model you’re using. In this course you’ll learn how to effectively prompt Gemini models. Gemini's multimodal capabilities also enable new approaches in AI application development, for example: - The Gemini library handles various video formats (MP4, MOV, MPEG), streamlining applications using these formats. - Large context window (up to 1 million tokens) enables processing of extensive content, like analyzing multiple 50-minute videos simultaneously. - Function calling feature integrates real-time data (e.g., current exchange rates) into model responses. The course demonstrates building multimodal applications with real-world examples including document analyzers that reason across text and graphs simultaneously, video content extractors that find and timestamp specific information from multiple hours of footage, and automated expense report systems processing receipt images while cross-referencing company policies. Sign up here:

Andrew Ng

74,060 views • 2 years ago

This is amazing! You can now create high-quality 3D Scenes from a single image using Multi-Instance Diffusion Models (MIDI) 🔥

This is amazing! You can now create high-quality 3D Scenes from a single image using Multi-Instance Diffusion Models (MIDI) 🔥

Gradio

41,940 views • 1 year ago