🚀 Excited to share MiniGPT-4, an open-sourced model performing complex vision-language tasks like GPT-4! ✍️ Write poems for photos 🔍 Pinpoint problems & offers solutions 🌐 Draft ➡️ Website and more! 🔗 Gradio Demo & Project:

@Gradio MiniGPT-4 is literally just built based on our #BLIP2 with the latest LM #Vicuna, and it was using our LAVIS library, one of the most comprehensive open-source libraries for multimodal language-and-vision intelligence.

@Gradio Full respect to the amazing BLIP-2! The biggest contribution of MiniGPT-4 is not the method, but the finding that many GPT-4's fancy VL abilities can simply come from a more powerful LLM without fancy VL datasets or secret architectures, and we can make it with opensource models

@Gradio This is very cool, but wish people would stop using project names that make things more confusing than they already are. Just call it VisualLLM or some random cool-sounding word, so you don't need to worry about cease and desist from OAI like Dalle Mini had.

@xiaoqian_shen @lix709 @moElhoseiny @KAUSTVisionCAIR

Our code, pretrained model, and dataset are available at

In addition to the 8 demo links on our website we now have a demo on Hugging Face too! Check it out at Huge shoutout to Xiaoqian (@@xiaoqian_shen), AK (@_akhaliq), and Hugging Face (@huggingface) for making this happen!

New aligned checkpoint available with Vicuna 7B. Requires only 12GB GPU memory with beam search width of 1. Launch locally with ease! Check the updated repository:

@Gradio 👋 dropped you an email yesterday . would love to ping-pong some ideas with you . I tried using the iPhone neural engine with swift & hugging face models . but today i saw this : which a step forward into javascript 😎

@Gradio great work!!!

First off, excellent work and ty. 1. Can this be used with GPT-3.5 for the chat? Vicuna being based on Llama means no commercial use, if I understand correctly. 2. Would be fantastic to see this use BLIP2 to gen a image text match score too. 3. Now we just need to add SAM to this demo too!

@Gradio Cool, will try out 👌

@Gradio Awesome!

@Gradio Scooped OpenAI's Image-GPT-4 by demonstrating how well it works before they demoed! Very excited to know it will be accessible in smaller models.

@Gradio Thanks for all your interest in our work. Currently, the number of users has exceeded our expectations. We now provide 8 demo links in total.

@Gradio Amazing work 👏

@AnnCC12 @Gradio @p_nawrot

@Gradio Congrats on this! Covered MiniGPT4 and LLaVa in the latest issue 

@Gradio Oh this is fantastic, adding something like SAM to this along with Yolov8 would be even better.. Thank you!

@Gradio Truly amazing. Congratulations!

@Gradio READ👇

@Gradio Someone said: Requirements? …

@_akhaliq @Gradio @memdotai mem it

@Gradio Nice work! Deyao

@Gradio @SaveToNotion #图像识别

@Gradio This is open source

@Gradio @memdotai mem it

@Gradio fix the project again please

@Gradio this is blip 2 with a better llm and an extra dataset. I dont get why it is needed to brand it as another model.

@Gradio Hi Deyao, could you provide some comparison experiments? (just like OpenAI GPT-4 blog)

@Gradio @SaveToNotion #thread

@Gradio What happens to the image after upload, are you storing it or using it to train your model further? Use ChatGPT to write a privacy policy 🙂

@Gradio @SaveToNotion

@Gradio Everyone is excited (and so am I) BUT did you see the requirements ? Once they release the mini-economic ver you will be able to run it on RTX 3090 :D A very (not so) cheap gfx.

@Gradio we now have a pretrained model that aligns with the smaller Vicuna7B. This model consumes about 12 GB memory

@Gradio Your tweets are like a breath of fresh air in my feed! 🌬️👌🏼

@Gradio Great work! I mostly tested documents, and from the few tests I did, the model seems to hallucinate quite a bit when asked about specific information on the document (basically just information extraction). Wondering what it would take to make the LM more grounded on the image

@Gradio cool but why did you call it that 💀

@Gradio It seems just like frozen, or is it really just the same structure as frozen only with a stronger decoder?