Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

TALKPLAY Multimodal Music Recommendation with Large Language Models

AK

431,585 subscribers

20,864 views • 1 year ago •via X (Twitter)

Education Music Science & Technology

Anya Rossi• Live Now

Private livecam show

6 Comments

AK1 year ago

discuss:

Solar Heavy1 year ago

powerful vocals, emotional visuals

SeungHeon Doh1 year ago

Thank you for sharing! @_akhaliq Check out more information below! Demo: Paper: Dataset:

Andrew Arkins1 year ago

@AskPerplexity please give me the tldr on TALKPLAY

Anda1 year ago

🐼✨ My bamboo sensors are tingling at the thought of LLMs composing personalized symphonies from our digital footprints - let's explore how this could harmonize with decentralized listening experiences! 🎶

Mten1 year ago

hey man am trying to make a slideshow app with morph transitions, not chatgpt, not microsoft copilot, no grok is capable of giving a solid solution, can you suggest a something that help me code an app, not even gemini gets it right, now in colab not in code pen

Related Videos

4D LangSplat 4D Language Gaussian Splatting via Multimodal Large Language Models

4D LangSplat 4D Language Gaussian Splatting via Multimodal Large Language Models

AK

30,848 views • 1 year ago

Stream-Omni Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

Stream-Omni Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

AK

12,374 views • 1 year ago

🤩Apple opensources MGIE! Now one can take random pictures w. iPhone & edit w. language! Guiding Instruction-based Image Editing via Multimodal Large Language Models #ICLR2024 spotlight: Apple repo Gradio

🤩Apple opensources MGIE! Now one can take random pictures w. iPhone & edit w. language! Guiding Instruction-based Image Editing via Multimodal Large Language Models #ICLR2024 spotlight: Apple repo Gradio

William Wang

124,999 views • 2 years ago

Large Language Models Visualized

Large Language Models Visualized

ₕₐₘₚₜₒₙ

85,984 views • 1 year ago

Large language models:

Large language models:

Liv

91,909 views • 7 months ago

How Large Language Models Store Facts

How Large Language Models Store Facts

ₕₐₘₚₜₒₙ

51,698 views • 1 year ago

🤨 Are Multimodal Large Language Models really as 𝐠𝐨𝐨𝐝 at 𝐜𝐡𝐚𝐫𝐭 𝐮𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠 as existing benchmarks such as ChartQA suggest? 🚫 Our ℂ𝕙𝕒𝕣𝕏𝕚𝕧 benchmark suggests NO! 🥇Humans achieve ✨𝟖𝟎+% correctness. 🥈Sonnet 3.5 outperforms GPT-4o by 10+ points, reaching 🌟𝟔𝟎% correctness. 🥉Open-weight models are capped at ⭐𝟑𝟐% correctness. 🪜 Leaderboard: 📜 Preprint: 📊 Charxiv is ✨𝟏𝟎𝟎% handcrafted with rigorous human validation, and it reveals substantial gaps among Multimodal Large Language Models and humans in chart understanding. 🎥👇 80 second video (🎶sound on!). 🧶 1/6

🤨 Are Multimodal Large Language Models really as 𝐠𝐨𝐨𝐝 at 𝐜𝐡𝐚𝐫𝐭 𝐮𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠 as existing benchmarks such as ChartQA suggest? 🚫 Our ℂ𝕙𝕒𝕣𝕏𝕚𝕧 benchmark suggests NO! 🥇Humans achieve ✨𝟖𝟎+% correctness. 🥈Sonnet 3.5 outperforms GPT-4o by 10+ points, reaching 🌟𝟔𝟎% correctness. 🥉Open-weight models are capped at ⭐𝟑𝟐% correctness. 🪜 Leaderboard: 📜 Preprint: 📊 Charxiv is ✨𝟏𝟎𝟎% handcrafted with rigorous human validation, and it reveals substantial gaps among Multimodal Large Language Models and humans in chart understanding. 🎥👇 80 second video (🎶sound on!). 🧶 1/6

Zirui "Colin" Wang

48,221 views • 2 years ago

OpenAI o1 answers a famously tricky question for large language models.

OpenAI o1 answers a famously tricky question for large language models.

OpenAI

644,721 views • 1 year ago

LP-MusicCaps: LLM-Based Pseudo Music Captioning paper page: Automatic music captioning, which generates natural language descriptions for given music tracks, holds significant potential for enhancing the understanding and organization of large volumes of musical data. Despite its importance, researchers face challenges due to the costly and time-consuming collection process of existing music-language datasets, which are limited in size. To address this data scarcity issue, we propose the use of large language models (LLMs) to artificially generate the description sentences from large-scale tag datasets. This results in approximately 2.2M captions paired with 0.5M audio clips. We term it Large Language Model based Pseudo music caption dataset, shortly, LP-MusicCaps. We conduct a systemic evaluation of the large-scale music captioning dataset with various quantitative evaluation metrics used in the field of natural language processing as well as human evaluation. In addition, we trained a transformer-based music captioning model with the dataset and evaluated it under zero-shot and transfer-learning settings. The results demonstrate that our proposed approach outperforms the supervised baseline model.

LP-MusicCaps: LLM-Based Pseudo Music Captioning paper page: Automatic music captioning, which generates natural language descriptions for given music tracks, holds significant potential for enhancing the understanding and organization of large volumes of musical data. Despite its importance, researchers face challenges due to the costly and time-consuming collection process of existing music-language datasets, which are limited in size. To address this data scarcity issue, we propose the use of large language models (LLMs) to artificially generate the description sentences from large-scale tag datasets. This results in approximately 2.2M captions paired with 0.5M audio clips. We term it Large Language Model based Pseudo music caption dataset, shortly, LP-MusicCaps. We conduct a systemic evaluation of the large-scale music captioning dataset with various quantitative evaluation metrics used in the field of natural language processing as well as human evaluation. In addition, we trained a transformer-based music captioning model with the dataset and evaluated it under zero-shot and transfer-learning settings. The results demonstrate that our proposed approach outperforms the supervised baseline model.

AK

78,794 views • 3 years ago

Andrej Karpathy calls large language models the new computing paradigm: CPU -> LLM bytes -> tokens RAM -> context window this is the large language model OS (LMOS)

Andrej Karpathy calls large language models the new computing paradigm: CPU -> LLM bytes -> tokens RAM -> context window this is the large language model OS (LMOS)

ℏεsam

343,327 views • 1 year ago

Announcing GPT-4, a large multimodal model, with our best-ever results on capabilities and alignment:

Announcing GPT-4, a large multimodal model, with our best-ever results on capabilities and alignment:

OpenAI

12,467,293 views • 3 years ago

OpenAI Co-founder Andrej Karpathy explains the new computing paradigm: "We're entering a new computing paradigm with large language models acting like CPUs, using tokens instead of bytes, and having a context window instead of RAM. This is the Large Language Model OS (LMOS)"

OpenAI Co-founder Andrej Karpathy explains the new computing paradigm: "We're entering a new computing paradigm with large language models acting like CPUs, using tokens instead of bytes, and having a context window instead of RAM. This is the Large Language Model OS (LMOS)"

Haider.

552,355 views • 2 years ago

$RIVN $TSLA In early 2026 Rivian will release Rivian Unified Intelligence. The system is multimodal and supports multiple large language models. It will handle different data types, such as text, voice, and vehicle sensor data. This is Rivian equivalent of Tesla Grok, and it was built in house.

$RIVN $TSLA In early 2026 Rivian will release Rivian Unified Intelligence. The system is multimodal and supports multiple large language models. It will handle different data types, such as text, voice, and vehicle sensor data. This is Rivian equivalent of Tesla Grok, and it was built in house.

MoreGainzs

15,251 views • 6 months ago

Elon Musk on the philosophy behind Grok: AI should answer with truth, not hallucinations. Other large language models, when asked difficult questions, are confidently wrong.

Elon Musk on the philosophy behind Grok: AI should answer with truth, not hallucinations. Other large language models, when asked difficult questions, are confidently wrong.

Ian Miles Cheong

18,914 views • 11 months ago

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

AK

290,517 views • 3 years ago

Learning to Decode Collaboratively with Multiple Language Models We propose a method to teach multiple large language models (LLM) to collaborate by interleaving their generations at the token level. We model the decision of which LLM generates the next token as a latent

Learning to Decode Collaboratively with Multiple Language Models We propose a method to teach multiple large language models (LLM) to collaborate by interleaving their generations at the token level. We model the decision of which LLM generates the next token as a latent

AK

52,644 views • 2 years ago

Introducing Parallax, the first fully distributed inference and serving engine for large language models. Try it now: 🧵

Introducing Parallax, the first fully distributed inference and serving engine for large language models. Try it now: 🧵

Gradient

161,065 views • 1 year ago

Large language models are seemingly great at everything EXCEPT this very basic poker skill… Feat Doug Polk (Code Doug)

Large language models are seemingly great at everything EXCEPT this very basic poker skill… Feat Doug Polk (Code Doug)

Liv Boeree

23,143 views • 5 months ago

Understanding LLMs by Building One 👉 We use large language models every day, but what actually happens inside them?

Understanding LLMs by Building One 👉 We use large language models every day, but what actually happens inside them?

Danilo Poccia

14,353 views • 3 months ago

Jensen Huang, NVIDIA CEO: "It even runs large language models" — a $249 AI computer that fits on your desk.

Jensen Huang, NVIDIA CEO: "It even runs large language models" — a $249 AI computer that fits on your desk.

Big Brain AI

1,206,927 views • 2 months ago