New 🤗 transformers release includes a very powerful Multimodel Large Language Model (MLLM) by Microsoft called KOSMOS-2! 🤩 The highlight of KOSMOS-2 is grounding, the model is *incredibly* accurate! 🌎 Play with the demo here 👉 But how does this model work? Let's take a look! 👀🧶

Grounding helps machine learning models relate to real-world examples. Including grounding makes models more performant by means of accuracy and robustness during inference. It also helps reduce the so-called "hallucinations" in language models.

In Kosmos-2, model is grounded to perform following tasks and is evaluated on 👇 - multimodal grounding & phrase grounding, e.g. localizing the object through natural language query - multimodal referring, e.g. describing object characteristics & location - perception-language tasks - language understanding and generation

The dataset used for grounding, called GRiT is also available on Hugging Face Hub 👉 Thanks to transformers integration, you can use KOSMOS-2 with few lines of code 🤩 See below! 👇

also big kudos to @ydshieh for implementing this in transformers ✨

Can Machine Learning beat the market? Check out this post on my free Substack where I share code and commentary for an XGBoost model and a Random Forest model that both deliver powerful performances.

multimodal* 🥲

@Microsoft Wow, its very fast on an A40

@Microsoft License?

@ClementDelangue @Microsoft 🤔 I could use this to improve I'm using the Blip model and is not bad but this looks like it could give more accurate results.

@Microsoft Yup KOSMOS-2 is awesome!

@Microsoft AI can understand a video at 3fps then !!!