Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

mapping out the visual language of film using a multimodal llm: i fed frames of a short film to a vision-language model and mapped out its ratings of surrealism and presence of human figure in each moment along the timeline. the result is an interactive playback interface based on... show more

Kat ⊷ the Poet Engineer

85,625 subscribers

94,879 görüntüleme • 1 yıl önce •via X (Twitter)

Sağlık & İyilik Sanat Bilim & Teknoloji

Anya Rossi• Live Now

Private livecam show

11 Yorum

Kat ⊷ the Poet Engineer profil fotoğrafı

Kat ⊷ the Poet Engineer1 yıl önce

frames with high visibility of face, hands, human figure are all rated pretty high on the gestural index (2nd row). the text underneath each moving frame is llm's explanation for its rating of level of human figure in that frame.

Kat ⊷ the Poet Engineer profil fotoğrafı

Kat ⊷ the Poet Engineer1 yıl önce

the short film used: maya deren's meshes of the afternoon, an experimental classic <3 i think the surrealism score also works pretty well, rating those frames with unusual imagery higher than others.

AssemblyAI profil fotoğrafı

AssemblyAI1 yıl önce

Announcing: Our most advanced speech-to-text model goes beyond accuracy to capture the real-world complexity of human conversation and deliver reliable, source-of-truth audio data. Explore Universal-2 updates 👇

David Marjanidze profil fotoğrafı

David Marjanidze1 yıl önce

I've never seen anything like the perspectives you share. It's truly original

Kieran profil fotoğrafı

Kieran1 yıl önce

I would honestly have to try it out, but everytime i see these types of interfaces, I get excited, then i try them and our human micromovments are just too much for most with out a lot of lag added on.

Kat ⊷ the Poet Engineer profil fotoğrafı

Kat ⊷ the Poet Engineer1 yıl önce

yea the model (media pipe) is definitely not perfect and it's been out a few years. i feel there's probably much better proprietary ones out there.

アメドオシナイケ profil fotoğrafı

アメドオシナイケ1 yıl önce

Your work is enchanting. You should teach a course

glia profil fotoğrafı

glia1 yıl önce

so cool!

Pixel Girl profil fotoğrafı

Pixel Girl1 yıl önce

😍

Ian profil fotoğrafı

Ian1 yıl önce

Minority Report UI vibes. Rad.

Mike Marinos profil fotoğrafı

Mike Marinos1 yıl önce

This was the post below yours :)

Benzer Videolar

What happens when we train the largest vision-language model and add in robot experiences? The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language. Website:

What happens when we train the largest vision-language model and add in robot experiences? The result is PaLM-E 🌴🤖, a 562-billion parameter, general-purpose, embodied visual-language generalist - across robotics, vision, and language. Website:

Danny Driess

1,272,600 görüntüleme • 3 yıl önce

Another example of the multiple TTS parallel pipelines pattern. Here's a voice AI agent that speaks both English and Arabic, using a specific model/voice for each language. These are PlayAI voices. The STT, TTS, and LLM inference is all running on Groq Inc. (The LLM is LLama 4 Maverick.)

Another example of the multiple TTS parallel pipelines pattern. Here's a voice AI agent that speaks both English and Arabic, using a specific model/voice for each language. These are PlayAI voices. The STT, TTS, and LLM inference is all running on Groq Inc. (The LLM is LLama 4 Maverick.)

kwindla

13,401 görüntüleme • 1 yıl önce

In 1969, the government of India's Films Division released a 3-minute animated short film titled Umbrella as part of its family planning campaign. The film used the simple visual metaphor of an umbrella to contrast the struggles of a large family with the stability and comfort of a smaller one

In 1969, the government of India's Films Division released a 3-minute animated short film titled Umbrella as part of its family planning campaign. The film used the simple visual metaphor of an umbrella to contrast the struggles of a large family with the stability and comfort of a smaller one

Dr. Lemma

1,092,033 görüntüleme • 8 ay önce

co-seeing art with machine: an interactive demo where i browse the visual responses of a multimodal llm to prompts about the painting, from “who’s in here?” and to “what’s unusual about this painting?” the model replies not just with words, but with visual annotations as well.

co-seeing art with machine: an interactive demo where i browse the visual responses of a multimodal llm to prompts about the painting, from “who’s in here?” and to “what’s unusual about this painting?” the model replies not just with words, but with visual annotations as well.

Kat ⊷ the Poet Engineer

37,359 görüntüleme • 1 yıl önce

Jia Zhangke made a short video thanking Film Comment voting CAUGHT BY THE TIDES as the fifth best film of the year, and calling for the renewal of film language for the evolving reality we live in

Jia Zhangke made a short video thanking Film Comment voting CAUGHT BY THE TIDES as the fifth best film of the year, and calling for the renewal of film language for the evolving reality we live in

Frank Yan

13,696 görüntüleme • 8 ay önce

RUSTLANDS - New interactive film OUT NOW @DreamFlareAI Follow a lone wanderer as she hunts for a fabled engine in the endless dunes of a steampunk desert wasteland. I've seriously been pushing the limits of AI for 2D animation on this one :) Be sure to check out the trailer & full interactive short film on the platform (link below)!

RUSTLANDS - New interactive film OUT NOW @DreamFlareAI Follow a lone wanderer as she hunts for a fabled engine in the endless dunes of a steampunk desert wasteland. I've seriously been pushing the limits of AI for 2D animation on this one :) Be sure to check out the trailer & full interactive short film on the platform (link below)!

Guillaume H.

20,380 görüntüleme • 1 yıl önce

Tired of your vision-language-action (VLA) model failing catastrophically in the presence of distractions? Check out BYOVLA: Bring Your Own VLA: a run-time intervention scheme that markedly improves performance with distractor objects and backgrounds.

Tired of your vision-language-action (VLA) model failing catastrophically in the presence of distractions? Check out BYOVLA: Bring Your Own VLA: a run-time intervention scheme that markedly improves performance with distractor objects and backgrounds.

Anirudha Majumdar

17,831 görüntüleme • 1 yıl önce

Holy shit, trump's approval ratings are in the TOILET, and a whopping 71% feel America is OUT OF CONTROL! That is a direct result of the out of control maniac "in charge."

Holy shit, trump's approval ratings are in the TOILET, and a whopping 71% feel America is OUT OF CONTROL! That is a direct result of the out of control maniac "in charge."

BrooklynDad_Defiant!☮️

291,019 görüntüleme • 6 ay önce

It's time to get out of stealth 👋 Today, we are launching Monogram and announcing our $40m seed round led by DST and Lux Capital. Monogram is the first AI app that was built around a visual interface, from the ground up. We created a technology that generates an entire user interface on the fly, in just a few seconds. Ask anything, and instead of staring at a wall of text, you get an interactive visual response.

It's time to get out of stealth 👋 Today, we are launching Monogram and announcing our $40m seed round led by DST and Lux Capital. Monogram is the first AI app that was built around a visual interface, from the ground up. We created a technology that generates an entire user interface on the fly, in just a few seconds. Ask anything, and instead of staring at a wall of text, you get an interactive visual response.

Eren Bali

1,328,898 görüntüleme • 1 ay önce

New short course Multimodal RAG: Chat with Videos, developed with Intel and taught by vasudevlal! In this course, you’ll work with LLaVA (Large Language and Vision Assistant), a Large Vision Language Model (LVLM) that can process both images and text. For example, given an image of a person doing a handstand on a skateboard at the beach, LLaVA doesn't just caption the scene, it’s able to predict possible outcomes, like the person losing balance or falling off. By understanding not just what's in a video frame, but what might happen next, your application can provide more insightful answers to questions about video. You'll build a full multimodal RAG pipeline that can chat about video content: - Use the BridgeTower model to create joint text-image embeddings in a 512-dimensional multimodal semantic space. - Learn video processing techniques to extract keyframes, generate transcripts using Whisper, and create captions. - Use the LanceDB vector database to store and retrieve high-dimensional multimodal embeddings. - Integrate the LLaVA model, combining CLIP's (Contrastive Language Image Pretraining) vision transformer with Llama, for advanced visual-textual reasoning. Your final system will ingest video data, generate embeddings for frames and text, perform similarity searches for relevant content, and use the retrieved multimodal context to inform LVLM-based response generation. The result is a system capable of answering nuanced questions about video content, effectively chatting about the video it has processed. Please sign up here!

New short course Multimodal RAG: Chat with Videos, developed with Intel and taught by vasudevlal! In this course, you’ll work with LLaVA (Large Language and Vision Assistant), a Large Vision Language Model (LVLM) that can process both images and text. For example, given an image of a person doing a handstand on a skateboard at the beach, LLaVA doesn't just caption the scene, it’s able to predict possible outcomes, like the person losing balance or falling off. By understanding not just what's in a video frame, but what might happen next, your application can provide more insightful answers to questions about video. You'll build a full multimodal RAG pipeline that can chat about video content: - Use the BridgeTower model to create joint text-image embeddings in a 512-dimensional multimodal semantic space. - Learn video processing techniques to extract keyframes, generate transcripts using Whisper, and create captions. - Use the LanceDB vector database to store and retrieve high-dimensional multimodal embeddings. - Integrate the LLaVA model, combining CLIP's (Contrastive Language Image Pretraining) vision transformer with Llama, for advanced visual-textual reasoning. Your final system will ingest video data, generate embeddings for frames and text, perform similarity searches for relevant content, and use the retrieved multimodal context to inform LVLM-based response generation. The result is a system capable of answering nuanced questions about video content, effectively chatting about the video it has processed. Please sign up here!

Andrew Ng

107,825 görüntüleme • 1 yıl önce

In an expansive ocean of vaporware, $RAI stands out as a beacon of innovation, offering you the opportunity to witness the dynamic process of augmenting our proprietary Large Language Model (LLM) with an astounding 680 million instruct parameters {Text generated by Reploy14b}

In an expansive ocean of vaporware, $RAI stands out as a beacon of innovation, offering you the opportunity to witness the dynamic process of augmenting our proprietary Large Language Model (LLM) with an astounding 680 million instruct parameters {Text generated by Reploy14b}

Reploy.ai

16,816 görüntüleme • 1 yıl önce

AI + robotics research is starting to pick up steam. We can now instruct Spot using natural language using Language-guided Skill Coordination (LSC). A user provides a natural language instruction: "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table" and the robot navigates to the location of the target objects and places them on the room table. Current versions of Spot rely on a fixed-set vocabulary that cannot generalize to diverse instructions. In this demo, the researchers presented a method that uses large language models to receive a free-form natural language instruction for object rearrangement, which it then executes. It combines: -A voice-to-text model that processes the instructions into text -An LLM that converts natural language instruction to call a library of skills -A perception module that provides ground truth locations and visual object detection And what your left with is a glimpse into what our future will hold: robots executing tasks from human voice prompts.

AI + robotics research is starting to pick up steam. We can now instruct Spot using natural language using Language-guided Skill Coordination (LSC). A user provides a natural language instruction: "Bring me the chocolates box, cereal box, and pill bottle, and put them on the bedroom table" and the robot navigates to the location of the target objects and places them on the room table. Current versions of Spot rely on a fixed-set vocabulary that cannot generalize to diverse instructions. In this demo, the researchers presented a method that uses large language models to receive a free-form natural language instruction for object rearrangement, which it then executes. It combines: -A voice-to-text model that processes the instructions into text -An LLM that converts natural language instruction to call a library of skills -A perception module that provides ground truth locations and visual object detection And what your left with is a glimpse into what our future will hold: robots executing tasks from human voice prompts.

AI Breakfast

33,725 görüntüleme • 3 yıl önce

Gladiator is a film of legend for countless reasons- not just because of its quality, but because the making of the film was improbable. For example, Oliver Reed, who played the gladiator trainer Proximo, died of a fatal heart attack on May 2, 1999, during a break in filming out in Malta. It’s just one of countless legendary stories to come out of the making of this film.

Gladiator is a film of legend for countless reasons- not just because of its quality, but because the making of the film was improbable. For example, Oliver Reed, who played the gladiator trainer Proximo, died of a fatal heart attack on May 2, 1999, during a break in filming out in Malta. It’s just one of countless legendary stories to come out of the making of this film.

Cinema Tweets

65,409 görüntüleme • 1 ay önce

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

AK

290,517 görüntüleme • 3 yıl önce

A little glimpse from the recording session of #Lokah. This piece of music lies at the heart of the film, beautifully brought to life by Anuradha Ma’am, Anila Rajeev, and the incredible choir. A special mention to #harinarayananbk , who created a unique language to perfectly match the Dominic Arun ‘s vision for this song. All tracks from the film, along with the full OST, will be out this week. Can’t wait to share it with you all. 💫

A little glimpse from the recording session of #Lokah. This piece of music lies at the heart of the film, beautifully brought to life by Anuradha Ma’am, Anila Rajeev, and the incredible choir. A special mention to #harinarayananbk , who created a unique language to perfectly match the Dominic Arun ‘s vision for this song. All tracks from the film, along with the full OST, will be out this week. Can’t wait to share it with you all. 💫

Jakes Bejoy

32,416 görüntüleme • 11 ay önce

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,657 görüntüleme • 3 yıl önce

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,353 görüntüleme • 3 yıl önce

‘Kryptonian Language Creation’ 💬 Meet Christine Schreyer, an assistant professor in the Irving K. Barber School of Arts & Science, and also a linguist & anthropologist at UBC's Okanagan campus who was a consultant on the film Man of Steel & created the Kryptonian language. #ManofSteel

‘Kryptonian Language Creation’ 💬 Meet Christine Schreyer, an assistant professor in the Irving K. Barber School of Arts & Science, and also a linguist & anthropologist at UBC's Okanagan campus who was a consultant on the film Man of Steel & created the Kryptonian language. #ManofSteel

𝗦𝘂𝗽𝗲𝗿𝗴𝗶𝗿𝗹 💫

36,343 görüntüleme • 1 yıl önce