Navigation überspringen
We introduce 🔥X-InstructBLIP🔥, a simple and effective scalable cross-modal framework to empower LLMs…
0:00 / 1:04

We introduce 🔥X-InstructBLIP🔥, a simple and effective scalable cross-modal framework to empower LLMs…

Caiming Xiong

@CaimingXiong8177 Follower

Auf X besuchen

37.476 Aufrufe

We introduce 🔥X-InstructBLIP🔥, a simple and effective scalable cross-modal framework to empower LLMs to handle tasks across modalities such as text, image, video, sound, and 3D. Web: ArXiv: Code:

5 Kommentare

Kommentare aus dem Originalbeitrag auf X (Twitter)

  • Profilbild von Caiming Xiong

    @CaimingXiong

    We extend InstructBLIP’s instruction-aware representations beyond images to 3D, audio, and video. Despite the lack of modality-specific pre-training, X-InstructBLIP achieves comparable performance to SoTA models on a variety of out-of-domain tasks and modalities.

    1 Like

  • Profilbild von Caiming Xiong

    @CaimingXiong

    Despite the lack of joint modality training and distinct frozen pre-trained encoders for each modality, X-InstructBLIP demonstrates emergent capabilities in cross-modal comprehension.

    1 Like

  • Profilbild von Caiming Xiong

    @CaimingXiong

    To evaluate its abilities we introduce a new Cross-modal Discriminative Reasoning benchmark (DisCRn): Given two distinct modality inputs, the model needs to select the entity that matches the property queried.

    1 Like

  • Profilbild von Caiming Xiong

    @CaimingXiong

    X-InstructBLIP outperforms a strong SoTA captioning baseline on the new DisCRn task by 6.3 and 3.2 points for image-3D and audio-video pairs respectively. Nevertheless, the task remains an open challenge.

    3 Likes

  • Profilbild von Caiming Xiong

    @CaimingXiong

    Thanks to all awesome collaborators: @artemispng, @Le_Xue01, @realNingYu, @LiJunnan0409, @dongxuli_, @JotyShafiq, @stanleyran, @silviocinguetta and @jcniebles

    4 Likes