Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

GutenOCR: A Grounded Vision-Language Front-End for Documents Fine-tunes Qwen2.5-VL-3B and 7B using a grounded OCR evaluation protocol, substantially improving region- and line-level OCR as well as text-detection recall

Niels Rogge

20,454 subscribers

45,811 views • 6 months ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Announcing llama-ocr – a free + open source OCR tool! It takes documents (images for now) & outputs markdown, and does really well for complex receipts, PDFs with tables/charts, ect... Powered by Llama 3.2 vision on Together AI & available on npm today!

Announcing llama-ocr – a free + open source OCR tool! It takes documents (images for now) & outputs markdown, and does really well for complex receipts, PDFs with tables/charts, ect... Powered by Llama 3.2 vision on Together AI & available on npm today!

Hassan

443,929 views • 1 year ago

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!

Fine-tune DeepSeek-OCR on your own language! (100% local) DeepSeek-OCR is a 3B-parameter vision model that achieves 97% precision while using 10× fewer vision tokens than text-based LLMs. It handles tables, papers, and handwriting without killing your GPU or budget. Why it matters: Most vision models treat documents as massive sequences of tokens, making long-context processing expensive and slow. DeepSeek-OCR uses context optical compression to convert 2D layouts into vision tokens, enabling efficient processing of complex documents. The best part? You can easily fine-tune it for your specific use case on a single GPU. I used Unsloth to run this experiment on Persian text and saw an 88.26% improvement in character error rate. ↳ Base model: 149% character error rate (CER) ↳ Fine-tuned model: 60% CER (57% more accurate) ↳ Training time: 60 steps on a single GPU Persian was just the test case. You can swap in your own dataset for any language, document type, or specific domain you're working with. I've shared the complete guide in the next tweet - all the code, notebooks, and environment setup ready to run with a single click. Everything is 100% open-source!

Akshay 🚀

126,122 views • 9 months ago

Mistral OCR 4 turned a handwritten calculus exam into clean LaTeX! We gave it a photo of a hand-written exam page. The model read the handwriting and rebuilt every formula into structured digital text Output: Time: 5.1s · Cost: $0.09 Formulas came through exactly right - the hard part was nailed. The graph, unfortunately, it didn’t redraw. But that’s the telling part: most OCR tools just dump the text and quietly drop the figure. OCR 4 caught the plot, boxed it, and tagged it as a chart. It doesn’t get redrawn, but it gets read and accounted for

Mistral OCR 4 turned a handwritten calculus exam into clean LaTeX! We gave it a photo of a hand-written exam page. The model read the handwriting and rebuilt every formula into structured digital text Output: Time: 5.1s · Cost: $0.09 Formulas came through exactly right - the hard part was nailed. The graph, unfortunately, it didn’t redraw. But that’s the telling part: most OCR tools just dump the text and quietly drop the figure. OCR 4 caught the plot, boxed it, and tagged it as a chart. It doesn’t get redrawn, but it gets read and accounted for

atomic.chat

418,922 views • 1 month ago

Baidu just open-sourced an OCR model that reads entire 40-page documents in one shot. It's called Unlimited-OCR. 3 billion parameters but only 500 million active during inference. Runs 100% locally on your machine. Why this matters: traditional OCR tools chop documents page by page. Tables that span two pages break. Reading order gets lost. Cross-page context disappears. Unlimited-OCR processes the whole document at once. 32K context window. Text, formulas, tables, reading order all preserved across pages. Output comes out as clean structured Markdown. → 93% accuracy on the standard benchmark. +6 points over the baseline. → Error rate stays below 0.11 even past 40 pages. → Multilingual out of the box. → 2.12 million downloads on Hugging Face last month. 14,600 GitHub stars. For context: Amazon Textract, Google Cloud Vision, and Azure Document Intelligence all charge per page. This runs locally for free.

Baidu just open-sourced an OCR model that reads entire 40-page documents in one shot. It's called Unlimited-OCR. 3 billion parameters but only 500 million active during inference. Runs 100% locally on your machine. Why this matters: traditional OCR tools chop documents page by page. Tables that span two pages break. Reading order gets lost. Cross-page context disappears. Unlimited-OCR processes the whole document at once. 32K context window. Text, formulas, tables, reading order all preserved across pages. Output comes out as clean structured Markdown. → 93% accuracy on the standard benchmark. +6 points over the baseline. → Error rate stays below 0.11 even past 40 pages. → Multilingual out of the box. → 2.12 million downloads on Hugging Face last month. 14,600 GitHub stars. For context: Amazon Textract, Google Cloud Vision, and Azure Document Intelligence all charge per page. This runs locally for free.

Vaibhav Sisinty

412,208 views • 16 days ago

NVIDIA just made AI detect objects 10x faster by deleting one step. It's called LocateAnything, and it removes the biggest bottleneck no one else was fixing in vision-language models. Normally a model builds each bounding box one coordinate token at a time. 100 objects means thousands of tokens before an answer. NVIDIA scrapped that: their Parallel Box Decoding predicts the whole box in a single forward pass, as one atomic unit. → 12.7 boxes/sec on one H100 → 10x faster than Qwen3-VL → +3.8% F1 on LVIS, accuracy up, not down → 3B params, runs on one consumer GPU Treating the box as one unit keeps its coordinates tied together, which is why accuracy climbed instead of falling. One model handles detection, GUI grounding, OCR, and document understanding, ready for computer-use agents, robotics, and document pipelines. 100% open source, weights, code, demo, and paper all live.

NVIDIA just made AI detect objects 10x faster by deleting one step. It's called LocateAnything, and it removes the biggest bottleneck no one else was fixing in vision-language models. Normally a model builds each bounding box one coordinate token at a time. 100 objects means thousands of tokens before an answer. NVIDIA scrapped that: their Parallel Box Decoding predicts the whole box in a single forward pass, as one atomic unit. → 12.7 boxes/sec on one H100 → 10x faster than Qwen3-VL → +3.8% F1 on LVIS, accuracy up, not down → 3B params, runs on one consumer GPU Treating the box as one unit keeps its coordinates tied together, which is why accuracy climbed instead of falling. One model handles detection, GUI grounding, OCR, and document understanding, ready for computer-use agents, robotics, and document pipelines. 100% open source, weights, code, demo, and paper all live.

Alvaro Cintas

201,346 views • 1 month ago

Ready to blast off to a whole new level? 🚀 🖱️The #ROGSpeedNova wireless performance streamlines each step within the data transmission protocol, as well as refining data packet size and frequency for optimized process efficiency. 👉🏻Learn more:

Ready to blast off to a whole new level? 🚀 🖱️The #ROGSpeedNova wireless performance streamlines each step within the data transmission protocol, as well as refining data packet size and frequency for optimized process efficiency. 👉🏻Learn more:

ROG Global

46,389 views • 2 years ago

this is the BEST vision language model I have ever tried! Aria is a new model by Rhymes.AI: a 25.3B multimodal model that can take image/video inputs 🤩 They release the model with Apache-2.0 license and fine-tuning scripts as well 👏 I tested it extensively, keep reading to learn more 🧶

this is the BEST vision language model I have ever tried! Aria is a new model by Rhymes.AI: a 25.3B multimodal model that can take image/video inputs 🤩 They release the model with Apache-2.0 license and fine-tuning scripts as well 👏 I tested it extensively, keep reading to learn more 🧶

merve

176,121 views • 1 year ago

Turkish Foreign Minister Hakan Fidan on Iran: The war in our region must end as soon as possible. For months, we have made great efforts to establish a negotiating table. Today as well, we say that diplomacy is the solution to problems, and we continue to work in that direction.

Turkish Foreign Minister Hakan Fidan on Iran: The war in our region must end as soon as possible. For months, we have made great efforts to establish a negotiating table. Today as well, we say that diplomacy is the solution to problems, and we continue to work in that direction.

Clash Report

28,130 views • 5 months ago

From our humble beginnings in 2011 to today's reimagined experience - watch as Goodnotes evolves through the years. What started as a simple digital paper app has transformed into a powerful platform for capturing ideas in any form. The new Goodnotes is here - with Whiteboards, Text Documents, and AI superpowers all in one unified place. No longer just Goodnotes 6, but simply Goodnotes - where your ideas come to life. Our journey continues with a fresh look and a bold new vision. Download now and experience the next chapter of note-taking.

From our humble beginnings in 2011 to today's reimagined experience - watch as Goodnotes evolves through the years. What started as a simple digital paper app has transformed into a powerful platform for capturing ideas in any form. The new Goodnotes is here - with Whiteboards, Text Documents, and AI superpowers all in one unified place. No longer just Goodnotes 6, but simply Goodnotes - where your ideas come to life. Our journey continues with a fresh look and a bold new vision. Download now and experience the next chapter of note-taking.

Goodnotes

33,191 views • 10 months ago

Have a little writing app I'm cooking that's designed just for my workflow. As you zoom out, every card flips to a one line summary so you can quickly scan and eval the argument one level up. Using Ollama local models to make these so they're quick and free. So simple, so useful.

Have a little writing app I'm cooking that's designed just for my workflow. As you zoom out, every card flips to a one line summary so you can quickly scan and eval the argument one level up. Using Ollama local models to make these so they're quick and free. So simple, so useful.

Maggie Appleton

37,940 views • 1 month ago

How can we address the scarcity of data required for specialized AI? Learn about Simula, a framework that reframes synthetic data generation as dataset-level mechanism design. By using reasoning to architect datasets from first principles, Simula enables fine-grained control over coverage, complexity, and quality. More →

How can we address the scarcity of data required for specialized AI? Learn about Simula, a framework that reframes synthetic data generation as dataset-level mechanism design. By using reasoning to architect datasets from first principles, Simula enables fine-grained control over coverage, complexity, and quality. More →

Google Research

142,546 views • 3 months ago

A new chapter starts today. As MegaETH mainnet approaches, Avon’s identity needs to reflect the system we’ve been building: a calm, transparent, and predictable place to lend or borrow. Our first brand was put together quickly during the early build phase. It let us move fast, but it never fully captured the level of clarity and focus we were aiming for. The new brand is built around those principles. Simple. Transparent. Grounded. A visual identity that matches Avon’s role as the credit and liquidity layer of MegaETH. It sets the tone for how lending should feel in a high-throughput, real-time environment: clear, controlled, and reliable. This is the foundation for what Avon is becoming as we head into mainnet.

Avon

56,449 views • 8 months ago

In the final chapters of the Book of Revelation, the end of the age is described as a massive confrontation between the forces of good and evil. The text says the kings and armies of the world gather for a final battle in a place called Armageddon. Yes, a place. According to the vision recorded by John of Patmos, strange signs appear in the sky, angels descend, and powerful figures emerge as the crazy battle between "godly" and worldly powers. The scene is filled with dramatic weird imagery including thunder, fire, and cosmic chaos as gods and earth seem to collide. The story goes on with Christ returning with heavenly armies to smash the forces against him, including the figures known as the Antichrist and the false prophet. After the battle the enemies of God are defeated and a period of judgment begins. The vision then moves toward a renewal of creation, where the old world passes away and a new reality emerges described as a new heaven and a new earth. The final chapters kinda show a symbolic picture of restoring justice after a time of horsesh*t and conflict. Are we on the cusp?

In the final chapters of the Book of Revelation, the end of the age is described as a massive confrontation between the forces of good and evil. The text says the kings and armies of the world gather for a final battle in a place called Armageddon. Yes, a place. According to the vision recorded by John of Patmos, strange signs appear in the sky, angels descend, and powerful figures emerge as the crazy battle between "godly" and worldly powers. The scene is filled with dramatic weird imagery including thunder, fire, and cosmic chaos as gods and earth seem to collide. The story goes on with Christ returning with heavenly armies to smash the forces against him, including the figures known as the Antichrist and the false prophet. After the battle the enemies of God are defeated and a period of judgment begins. The vision then moves toward a renewal of creation, where the old world passes away and a new reality emerges described as a new heaven and a new earth. The final chapters kinda show a symbolic picture of restoring justice after a time of horsesh*t and conflict. Are we on the cusp?

Jason Wilde

14,651 views • 4 months ago

We’re excited to introduce Text-to-LoRA: a Hypernetwork that generates task-specific LLM adapters (LoRAs) based on a text description of the task. Catch our presentation at #ICML2025! Paper: Code: Biological systems are capable of rapid adaptation, given limited sensory cues. For example, our human visual system can quickly adapt and tune its light sensitivity to our surroundings. While modern LLMs exhibit a wide variety of capabilities and knowledge, they remain rigid when adding task-specific capabilities. Traditionally, customizing these models requires gathering large datasets and performing often expensive, time-consuming fine-tuning for specific applications. To bypass these limitations, Text-to-LoRA (T2L) meta-learns a “hypernetwork” that takes in a text description of a desired task, as a prompt, and generates a task-specific LoRA that performs well on the task. In our experiments, we show that T2L can encode hundreds of existing LoRA adapters. While the compression is lossy, T2L maintains the performance of task-specifically tuned LoRA adapters. We also show that T2L can even generalize to unseen tasks given a natural language description of the tasks. Importantly, Text-to-LoRA is parameter-efficient. It generates LoRAs in a single, inexpensive step, based solely on a simple text description of the task. This approach is a step towards dramatically lowering the technical and computational barriers, allowing non-technical users to specialize foundation models using plain language, rather than needing deep technical expertise or large compute resources.

We’re excited to introduce Text-to-LoRA: a Hypernetwork that generates task-specific LLM adapters (LoRAs) based on a text description of the task. Catch our presentation at #ICML2025! Paper: Code: Biological systems are capable of rapid adaptation, given limited sensory cues. For example, our human visual system can quickly adapt and tune its light sensitivity to our surroundings. While modern LLMs exhibit a wide variety of capabilities and knowledge, they remain rigid when adding task-specific capabilities. Traditionally, customizing these models requires gathering large datasets and performing often expensive, time-consuming fine-tuning for specific applications. To bypass these limitations, Text-to-LoRA (T2L) meta-learns a “hypernetwork” that takes in a text description of a desired task, as a prompt, and generates a task-specific LoRA that performs well on the task. In our experiments, we show that T2L can encode hundreds of existing LoRA adapters. While the compression is lossy, T2L maintains the performance of task-specifically tuned LoRA adapters. We also show that T2L can even generalize to unseen tasks given a natural language description of the tasks. Importantly, Text-to-LoRA is parameter-efficient. It generates LoRAs in a single, inexpensive step, based solely on a simple text description of the task. This approach is a step towards dramatically lowering the technical and computational barriers, allowing non-technical users to specialize foundation models using plain language, rather than needing deep technical expertise or large compute resources.

Sakana AI

403,159 views • 1 year ago

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation

The Hidden Language of Diffusion Models paper page: tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation

AK

41,746 views • 3 years ago

🧬 We have many foundation models or language models for DNAs, but can we control them? We introduce Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RL — a reinforcement learning framework for controllable cis-regulatory sequence generation. Paper: Code: 🔬What’s the challenge? Designing regulatory DNA that is both highly expressive in target cell types and inactive in others is essential for synthetic biology, gene therapy, and precision medicine. Yet, controlling these trade-offs is challenging due to sparse, sequence-level rewards and biological constraints. 🔥Why Ctrl-DNA? Ctrl-DNA fine-tunes pre-trained DNA language models using a value model free, Lagrangian-guided RL framework, enabling flexible and customizable constraint optimization. Users can define application-specific thresholds across cell types, balancing expression strength with specificity. ✅ Maximize target-cell expression ✅ Constrain off-target activity under user-defined thresholds ✅ Preserve cell-type-specific TF motif structure Benchmarked on human enhancer and promoter datasets, Ctrl-DNA consistently outperforms prior methods, achieving stronger specificity, higher fitness, and more biologically grounded sequence generation — all with direct control over regulatory trade-offs. Shoutout to the PhD students Xingyu Chen (Xingyu Chen ) and Rex Ma (Rex Ma) for their amazing work leading this project!

🧬 We have many foundation models or language models for DNAs, but can we control them? We introduce Ctrl-DNA: Controllable Cell-Type-Specific Regulatory DNA Design via Constrained RL — a reinforcement learning framework for controllable cis-regulatory sequence generation. Paper: Code: 🔬What’s the challenge? Designing regulatory DNA that is both highly expressive in target cell types and inactive in others is essential for synthetic biology, gene therapy, and precision medicine. Yet, controlling these trade-offs is challenging due to sparse, sequence-level rewards and biological constraints. 🔥Why Ctrl-DNA? Ctrl-DNA fine-tunes pre-trained DNA language models using a value model free, Lagrangian-guided RL framework, enabling flexible and customizable constraint optimization. Users can define application-specific thresholds across cell types, balancing expression strength with specificity. ✅ Maximize target-cell expression ✅ Constrain off-target activity under user-defined thresholds ✅ Preserve cell-type-specific TF motif structure Benchmarked on human enhancer and promoter datasets, Ctrl-DNA consistently outperforms prior methods, achieving stronger specificity, higher fitness, and more biologically grounded sequence generation — all with direct control over regulatory trade-offs. Shoutout to the PhD students Xingyu Chen (Xingyu Chen ) and Rex Ma (Rex Ma) for their amazing work leading this project!

Bo Wang

30,719 views • 1 year ago

We’re thrilled to announce our partnership with @Reployai , a cutting-edge AI ecosystem that bridges developers and projects to the blockchain using natural language processing (NLP). This collaboration aligns with Aurk AI’s mission to revolutionize accessibility and functionality in AI and blockchain. Through our partnership, Aurk AI and Rai will deliver next-level solutions for AI and blockchain enthusiasts, ensuring scalability, usability, and innovation at every step. Stay tuned as we unlock new possibilities with this powerful collaboration!

We’re thrilled to announce our partnership with @Reployai , a cutting-edge AI ecosystem that bridges developers and projects to the blockchain using natural language processing (NLP). This collaboration aligns with Aurk AI’s mission to revolutionize accessibility and functionality in AI and blockchain. Through our partnership, Aurk AI and Rai will deliver next-level solutions for AI and blockchain enthusiasts, ensuring scalability, usability, and innovation at every step. Stay tuned as we unlock new possibilities with this powerful collaboration!

AURK AI

44,263 views • 1 year ago

This is exactly the kind of symbol that Moscow has hated for centuries: a community of memory. The presidents of Poland, Lithuania and Ukraine, together with free Belarusians, standing at the Rossa Cemetery in Vilnius, in front of the chapel of the January Uprising fighters. This image strikes at the very core of Russia’s imperial narrative. It shows a shared history of resistance to empire, a continuity of struggle from the 19th century to today, and an alternative vision of the region, not as a “sphere of influence”, but as a community of nations with their own memory and dignity. 🇵🇱🇱🇹🇺🇦

This is exactly the kind of symbol that Moscow has hated for centuries: a community of memory. The presidents of Poland, Lithuania and Ukraine, together with free Belarusians, standing at the Rossa Cemetery in Vilnius, in front of the chapel of the January Uprising fighters. This image strikes at the very core of Russia’s imperial narrative. It shows a shared history of resistance to empire, a continuity of struggle from the 19th century to today, and an alternative vision of the region, not as a “sphere of influence”, but as a community of nations with their own memory and dignity. 🇵🇱🇱🇹🇺🇦

Marcin Czapliński

192,746 views • 6 months ago

Show-o One Single Transformer to Unify Multimodal Understanding and Generation discuss: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model.

Show-o One Single Transformer to Unify Multimodal Understanding and Generation discuss: We present a unified transformer, i.e., Show-o, that unifies multimodal understanding and generation. Unlike fully autoregressive models, Show-o unifies autoregressive and (discrete) diffusion modeling to adaptively handle inputs and outputs of various and mixed modalities. The unified model flexibly supports a wide range of vision-language tasks including visual question-answering, text-to-image generation, text-guided inpainting/extrapolation, and mixed-modality generation. Across various benchmarks, it demonstrates comparable or superior performance to existing individual models with an equivalent or larger number of parameters tailored for understanding or generation. This significantly highlights its potential as a next-generation foundation model.

AK

124,048 views • 1 year ago

We're excited to announce that Omni Spot Swap by ApeX Protocol is now officially live on #ApeXOmni, bringing a seamless, chain-agnostic spot trading experience to #DeFi: Effortlessly trade any assets across major blockchains like Ethereum, BNB Chain, Solana and Base — all using $USDT as the unified currency for both trades and gas fees. ➜ No need for bridging or wrapping! ➜ No need to hold multiple tokens to cover gas fees on different chains! Enjoy efficient, seamless and secure spot trading with CEX-level speed and protection against front-running. ➡️ Learn More: Try out Omni Spot Swap today — Make your first trade and share your feedback to earn a USDT bonus and a Welcome Mystery Box! ➡️Learn More:

We're excited to announce that Omni Spot Swap by ApeX Protocol is now officially live on #ApeXOmni, bringing a seamless, chain-agnostic spot trading experience to #DeFi: Effortlessly trade any assets across major blockchains like Ethereum, BNB Chain, Solana and Base — all using $USDT as the unified currency for both trades and gas fees. ➜ No need for bridging or wrapping! ➜ No need to hold multiple tokens to cover gas fees on different chains! Enjoy efficient, seamless and secure spot trading with CEX-level speed and protection against front-running. ➡️ Learn More: Try out Omni Spot Swap today — Make your first trade and share your feedback to earn a USDT bonus and a Welcome Mystery Box! ➡️Learn More:

ApeX Protocol

39,087 views • 1 year ago