Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

We just released 3 million samples of high quality vision language model training dataset for use cases such as: 📄 optical character recognition (OCR) 📊 visual question answering (VQA) 📝 captioning 🤗 Learn more: 📥 Download:

NVIDIA AI Developer

111,550 subscribers

95,858 views • 1 year ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

23 Comments

Energy Pulse1 year ago

This is gold for anyone building multimodal models, Bravo NVIDIA!👏

Furkan Gözükara1 year ago

Just stop selling low vram from ridiculous prices I paid 32 gb vram 5090, 4000 usd

Vaibhav (VB) Srivastav1 year ago

brilliant release!

PageIndex1 year ago

Most OCR tools only extract each page independently, losing the broader document context and hierarchy. PageIndex OCR is the first long-context OCR model that preserves the document’s global hierarchical structure—outperforming Mistral OCR.

Cyril Zakka, MD1 year ago

Amazing!

Quentin Lhoest 🤗1 year ago

So cool ! It would be amazing to improve the structure to show the images in the Dataset Viewer, happy to help :)

Bo1 year ago

cc @michael_g_u please fwd to G & A 😅

Alex Veremeyenko1 year ago

this dataset drop is a game changer for multimodal projects, excited to see what the community builds with it

Breakpoint1 year ago

🔥🔥🔥

Nathan1 year ago

nice !

無限 💀 acc/acc1 year ago

🤤 Data.

Aero - AI & Crypto1 year ago

Data will fuel open-source. @Alibaba_Qwen your turn. @AIatMeta @Zai_org

Hardy Chen 🏃 COLM20261 year ago

Thanks for the great sharing! But it seems that the ocr annotations and images are not paired. Did I miss anything?

Ammar Fahmy1 year ago

@huggingface Awsm 👌

jr1 year ago

thank you for your work!! Do all the images uploaded to HF yet? I see some of the folders missing.

Robert Youssef1 year ago

that's a serious dataset drop. gonna make some waves in multimodal projects. nice to see real value for the community

ZIAISTAN1 year ago

Excellent can it be used to train from scratch?

Numan1 year ago

@huggingface wowowow

Iqra Saifi1 year ago

Wowww

seekgpt1 year ago

Great 👍

Caleb1 year ago

Amazing! 🤩

Himanshu Kumar1 year ago

Large datasets are key for robust AI. This opens exciting new avenues for computer vision research. What innovative applications will this unlock?

Moon 🎑1 year ago

Really helpful for vision based model architecture.

Related Videos

Introducing Meta Perception Language Model (PLM): an open & reproducible vision-language model tackling challenging visual tasks. Learn more about how PLM can help the open source community build more capable computer vision systems. Read the research paper, and download the code and dataset:

Introducing Meta Perception Language Model (PLM): an open & reproducible vision-language model tackling challenging visual tasks. Learn more about how PLM can help the open source community build more capable computer vision systems. Read the research paper, and download the code and dataset:

AI at Meta

94,534 views • 1 year ago

From recognizing text to understanding documents. 📄 Meet Xiaomi-OCR-0, an open-source document understanding model with just 0.8B parameters. It brings document parsing, key information extraction, and OCR-oriented visual question answering into one lightweight model. It scores 95.24 on Real5-OmniDocBench and 83.2 on average across five OCR-oriented VQA benchmarks. On OmniDocBench v1.6, Xiaomi-OCR-0 ranks first in document parsing performance among models with fewer than 1B parameters. Model weights, inference code, an interactive demo, MCP service, and Agent Skill are now open source. 👇 🤗 Huggingface: 🔗 GitHub: 📄 Demo: #XiaomiAI #OCR #OpenSource

From recognizing text to understanding documents. 📄 Meet Xiaomi-OCR-0, an open-source document understanding model with just 0.8B parameters. It brings document parsing, key information extraction, and OCR-oriented visual question answering into one lightweight model. It scores 95.24 on Real5-OmniDocBench and 83.2 on average across five OCR-oriented VQA benchmarks. On OmniDocBench v1.6, Xiaomi-OCR-0 ranks first in document parsing performance among models with fewer than 1B parameters. Model weights, inference code, an interactive demo, MCP service, and Agent Skill are now open source. 👇 🤗 Huggingface: 🔗 GitHub: 📄 Demo: #XiaomiAI #OCR #OpenSource

Xiaomi Tech

154,502 views • 15 days ago

Florence-2, the new vision foundation model by Microsoft, can now run 100% locally in your browser on WebGPU, thanks to Transformers.js! 🤗🤯 It supports tasks like image captioning, optical character recognition, object detection, and many more! 😍 WOW! Demo (+ source code) 👇

Florence-2, the new vision foundation model by Microsoft, can now run 100% locally in your browser on WebGPU, thanks to Transformers.js! 🤗🤯 It supports tasks like image captioning, optical character recognition, object detection, and many more! 😍 WOW! Demo (+ source code) 👇

Xenova

88,782 views • 2 years ago

NVIDIA drop a 3B vision-language model for fast, high-quality visual grounding in real time accurately. - Parallel box decoding - 10× faster than Qwen3-VL - Trained on 138M queries/785M boxes - GUI, OCR, and document layout, dense detection - Open source Useful for computer-use agents and Physical AI,

NVIDIA drop a 3B vision-language model for fast, high-quality visual grounding in real time accurately. - Parallel box decoding - 10× faster than Qwen3-VL - Trained on 138M queries/785M boxes - GUI, OCR, and document layout, dense detection - Open source Useful for computer-use agents and Physical AI,

Md Ismail Šojal 🕷️

134,048 views • 1 month ago

📣 Microsoft Research releases Florence-VL, a new family of MLLMs powered by the generative vision foundation model Florence-2. Achieves significant improvements in general VQA, perception, hallucination, OCR, Chart, knowledge-intensive understanding, and more🔥Learn more👇

📣 Microsoft Research releases Florence-VL, a new family of MLLMs powered by the generative vision foundation model Florence-2. Achieves significant improvements in general VQA, perception, hallucination, OCR, Chart, knowledge-intensive understanding, and more🔥Learn more👇

Gradio

14,454 views • 1 year ago

Meta FAIR recently released the Seamless Interaction Dataset, the largest known high-quality video dataset of its kind, with: 4,000+ diverse participants 4,000+ hours of footage 65k+ interactions 5,000+ annotated samples This dataset of full-body, in-person, face-to-face interaction videos represents a crucial stepping stone to understanding and modeling how people communicate and behave when they’re together—advancing AI's ability to generate more natural conversations and human-like gestures. Download the dataset on Hugging Face: Learn more about the dataset:

Meta FAIR recently released the Seamless Interaction Dataset, the largest known high-quality video dataset of its kind, with: 4,000+ diverse participants 4,000+ hours of footage 65k+ interactions 5,000+ annotated samples This dataset of full-body, in-person, face-to-face interaction videos represents a crucial stepping stone to understanding and modeling how people communicate and behave when they’re together—advancing AI's ability to generate more natural conversations and human-like gestures. Download the dataset on Hugging Face: Learn more about the dataset:

AI at Meta

24,133 views • 1 year ago

LP-MusicCaps: LLM-Based Pseudo Music Captioning paper page: Automatic music captioning, which generates natural language descriptions for given music tracks, holds significant potential for enhancing the understanding and organization of large volumes of musical data. Despite its importance, researchers face challenges due to the costly and time-consuming collection process of existing music-language datasets, which are limited in size. To address this data scarcity issue, we propose the use of large language models (LLMs) to artificially generate the description sentences from large-scale tag datasets. This results in approximately 2.2M captions paired with 0.5M audio clips. We term it Large Language Model based Pseudo music caption dataset, shortly, LP-MusicCaps. We conduct a systemic evaluation of the large-scale music captioning dataset with various quantitative evaluation metrics used in the field of natural language processing as well as human evaluation. In addition, we trained a transformer-based music captioning model with the dataset and evaluated it under zero-shot and transfer-learning settings. The results demonstrate that our proposed approach outperforms the supervised baseline model.

LP-MusicCaps: LLM-Based Pseudo Music Captioning paper page: Automatic music captioning, which generates natural language descriptions for given music tracks, holds significant potential for enhancing the understanding and organization of large volumes of musical data. Despite its importance, researchers face challenges due to the costly and time-consuming collection process of existing music-language datasets, which are limited in size. To address this data scarcity issue, we propose the use of large language models (LLMs) to artificially generate the description sentences from large-scale tag datasets. This results in approximately 2.2M captions paired with 0.5M audio clips. We term it Large Language Model based Pseudo music caption dataset, shortly, LP-MusicCaps. We conduct a systemic evaluation of the large-scale music captioning dataset with various quantitative evaluation metrics used in the field of natural language processing as well as human evaluation. In addition, we trained a transformer-based music captioning model with the dataset and evaluated it under zero-shot and transfer-learning settings. The results demonstrate that our proposed approach outperforms the supervised baseline model.

AK

78,794 views • 3 years ago

Blender Studio released Project Storm, a high-quality, animation-ready character rig designed for Blender 5.0. Learn how it was made and download here:

Blender Studio released Project Storm, a high-quality, animation-ready character rig designed for Blender 5.0. Learn how it was made and download here:

80 LEVEL

50,593 views • 11 months ago

🚨 Just released: MIRIAD, a million-scale medical QA dataset to ground LLMs in reliable medical knowledge. 5.8M question-answer pairs, each distilled from peer-reviewed literature! 🔥 That's structured, high-quality data built for medical AI. 🧵 ↓

🚨 Just released: MIRIAD, a million-scale medical QA dataset to ground LLMs in reliable medical knowledge. 5.8M question-answer pairs, each distilled from peer-reviewed literature! 🔥 That's structured, high-quality data built for medical AI. 🧵 ↓

Charly Wargnier

48,127 views • 1 year ago

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

DeepLearning.AI

16,779 views • 1 year ago

Today we presented our work on Action Recognition with 3D pose and tracking at #CVPR. We also released our code, models, and dataset of 1.5 million human trajectories. Code: Project Page:

Today we presented our work on Action Recognition with 3D pose and tracking at #CVPR. We also released our code, models, and dataset of 1.5 million human trajectories. Code: Project Page:

Jathushan Rajasegaran

132,016 views • 3 years ago

Very excited to introduce Paperclip Datasets! Curate your own datasets from millions of papers in minutes. Just describe the dataset you want to build! As example, we release the largest dataset of 20k half-life human measurements curated from 2400+ drugs💊 🖇️ Try it: 📝 Blog: 📊 Half-life dataset:

Very excited to introduce Paperclip Datasets! Curate your own datasets from millions of papers in minutes. Just describe the dataset you want to build! As example, we release the largest dataset of 20k half-life human measurements curated from 2400+ drugs💊 🖇️ Try it: 📝 Blog: 📊 Half-life dataset:

James Zou

71,480 views • 10 days ago

Bengaluru-based startup Sarvam AI is drawing global attention with its latest AI models, Sarvam Vision and Bulbul V3. The company claims its technology has outperformed global giants like Google Gemini and ChatGPT in key optical character recognition (OCR) benchmarks. If verified, the breakthrough could mark a major moment for India’s growing AI ecosystem. What makes Sarvam Vision different? And how does Bulbul V3 push the boundaries of OCR performance? #SarvamAI

Bengaluru-based startup Sarvam AI is drawing global attention with its latest AI models, Sarvam Vision and Bulbul V3. The company claims its technology has outperformed global giants like Google Gemini and ChatGPT in key optical character recognition (OCR) benchmarks. If verified, the breakthrough could mark a major moment for India’s growing AI ecosystem. What makes Sarvam Vision different? And how does Bulbul V3 push the boundaries of OCR performance? #SarvamAI

NDTV

51,372 views • 9 months ago

Introducing Objaverse-XL, an open dataset of over 10 million 3D objects! With it, we train Zero123-XL, a foundation model for 3D, observing incredible 3D generalization abilities: 🧵👇 📝 Paper:

Introducing Objaverse-XL, an open dataset of over 10 million 3D objects! With it, we train Zero123-XL, a foundation model for 3D, observing incredible 3D generalization abilities: 🧵👇 📝 Paper:

Matt Deitke

368,934 views • 3 years ago

Today we’re introducing Scribe v2: the most accurate transcription model ever released. While Scribe v2 Realtime is optimized for ultra low latency and agents use cases, Scribe v2 is built for batch transcription, subtitling, and captioning at scale.

Today we’re introducing Scribe v2: the most accurate transcription model ever released. While Scribe v2 Realtime is optimized for ultra low latency and agents use cases, Scribe v2 is built for batch transcription, subtitling, and captioning at scale.

ElevenLabs

558,355 views • 10 months ago

🚀 Meta FAIR is releasing several new research artifacts on our road to advanced machine intelligence (AMI). These latest advancements are transforming our understanding of perception. 1️⃣ Meta Perception Encoder: A large-scale vision encoder that excels across several image & video tasks. 2️⃣ Meta Perception Language Model: A fully open & reproducible vision-language model designed to tackle visual recognition tasks. 3️⃣ Meta Locate 3D: An end-to-end model for accurate object localization in 3D environments. 4️⃣ Releasing model weights for our 8B-parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. 5️⃣Collaborative Reasoner: A framework for evaluating & improving collaborative reasoning skills in language models. Download the code, datasets, and research papers and learn more about how these artifacts are paving the way for more efficient and accurate AI systems.➡️

🚀 Meta FAIR is releasing several new research artifacts on our road to advanced machine intelligence (AMI). These latest advancements are transforming our understanding of perception. 1️⃣ Meta Perception Encoder: A large-scale vision encoder that excels across several image & video tasks. 2️⃣ Meta Perception Language Model: A fully open & reproducible vision-language model designed to tackle visual recognition tasks. 3️⃣ Meta Locate 3D: An end-to-end model for accurate object localization in 3D environments. 4️⃣ Releasing model weights for our 8B-parameter Dynamic Byte Latent Transformer, an alternative to traditional tokenization methods with the potential to redefine the standards for language model efficiency and reliability. 5️⃣Collaborative Reasoner: A framework for evaluating & improving collaborative reasoning skills in language models. Download the code, datasets, and research papers and learn more about how these artifacts are paving the way for more efficient and accurate AI systems.➡️

AI at Meta

163,663 views • 1 year ago

Let's talk parsing charts 📊📈. Last week we released ParseBench, the first document OCR benchmark for AI agents. New in ParseBench: ChartDataPointMatch. Most document look at a chart and OCR the caption. Agents need the actual numbers. That's the gap between "OCR'd the text around the chart" and "actually read the chart." More about ParseBench, the GitHub code, Hugging Face dataset, and scientific paper→

Let's talk parsing charts 📊📈. Last week we released ParseBench, the first document OCR benchmark for AI agents. New in ParseBench: ChartDataPointMatch. Most document look at a chart and OCR the caption. Agents need the actual numbers. That's the gap between "OCR'd the text around the chart" and "actually read the chart." More about ParseBench, the GitHub code, Hugging Face dataset, and scientific paper→

LlamaIndex 🦙

13,987 views • 6 months ago

What happens when we train the largest vision-language model and add in robot experiences? The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language. Website:

What happens when we train the largest vision-language model and add in robot experiences? The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language. Website:

Danny Driess

1,272,956 views • 3 years ago

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

250,159 views • 3 years ago