Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

AudioLDM 2: text to audio/music/speech generation Code/models/paper coming soon

_akhaliq's profile picture

AK

508,722 subscribers

202,456 Aufrufe • vor 3 Jahren •via X (Twitter)

BildungWissenschaft & TechnologieMusik
LIVE

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von David Marx (@digthatdata.bsky.social)
David Marx (@digthatdata.bsky.social)vor 3 Jahren

lol the heavy bias for weird ASMR mouth noises when "soft whispers" showed up in the prompt

Profilbild von Cristian Peñas ░░░░░░░░
Cristian Peñas ░░░░░░░░vor 3 Jahren

I'm excited to start making my ideal playlist with this tools!

Profilbild von Aiifi.ai
Aiifi.aivor 3 Jahren

Can't wait to try this out but do you mind if I ask what sets this tool apart from the other AI text to media tools? Its exciting times for AI and #ArtificialIntelligence

Profilbild von MET AMS
MET AMSvor 3 Jahren

Incredibly exciting stuff 🔉

Profilbild von CrazyTimes
CrazyTimesvor 3 Jahren

Great music paper coming out this year, soon Ai will generate 90% of the media content we consume.

Profilbild von Furkan Gözükara
Furkan Gözükaravor 3 Jahren

But how many seconds That is the key And the bitrate and quality

Profilbild von Earl Cameron
Earl Cameronvor 3 Jahren

Cc audio webui

Profilbild von Karthik Sharma
Karthik Sharmavor 3 Jahren

This is sounding way between than Meta's Music LM . Love the speech generation the end. We're living in great times

Profilbild von JoJo 99AI
JoJo 99AIvor 3 Jahren

Exciting! Can't wait.

Profilbild von SGM
SGMvor 3 Jahren

Is this similar to google soundstorm?

Ähnliche Videos

try out the Gradio Demo for AudioLDM: Text-to-Audio Generation with Latent Diffusion Models on Hugging Face demo:
1:36

try out the Gradio Demo for AudioLDM: Text-to-Audio Generation with Latent Diffusion Models on Hugging Face demo:

AK

82,138 Aufrufe • vor 3 Jahren

models under openAI: • text-to-text (gpt-2) • text-to-text (gpt-3) • text-to-code (codex) • text-to-text (chatGPT) • text-to-music (jukebox) • speech-to-text (whisper) • text-to-image (DALLE-E 2) #generativeai #chatgpt
0:30

models under openAI: • text-to-text (gpt-2) • text-to-text (gpt-3) • text-to-code (codex) • text-to-text (chatGPT) • text-to-music (jukebox) • speech-to-text (whisper) • text-to-image (DALLE-E 2) #generativeai #chatgpt

aaronsim

67,722 Aufrufe • vor 3 Jahren

The winner of Lovable's weekend competition: Kolbo ai - A powerful tool to help make all sorts of social media content with AI Features of the winning app: - Supabase for backend - Project-based organization system - OpenAI for text & image generation - Anthropic for text generation - Google Gemini for text generation - Midjourney for image generation - for image generation - Text-to-speech - Speech-to-text - Stripe for payments - mu for music generation Built by Zohar Vanunu 👇
5:29

The winner of Lovable's weekend competition: Kolbo ai - A powerful tool to help make all sorts of social media content with AI Features of the winning app: - Supabase for backend - Project-based organization system - OpenAI for text & image generation - Anthropic for text generation - Google Gemini for text generation - Midjourney for image generation - for image generation - Text-to-speech - Speech-to-text - Stripe for payments - mu for music generation Built by Zohar Vanunu 👇

Lovable

35,891 Aufrufe • vor 1 Jahr

Wow! New Speech to Speech model - Fish Agent v0.1 3B by Fish Audio 🔥 > Trained on 700K hours of multilingual audio > Continue-pretrained version of Qwen-2.5-3B-Instruct for 200B audio & text tokens > Zero-shot voice cloning > Text + audio input/ Audio output > Ultra-fast inference w/ 200ms TTFA > Models on the Hub & Finetuning code on its way! 🚀 What an amazing time to be alive 🤗
1:08

Wow! New Speech to Speech model - Fish Agent v0.1 3B by Fish Audio 🔥 > Trained on 700K hours of multilingual audio > Continue-pretrained version of Qwen-2.5-3B-Instruct for 200B audio & text tokens > Zero-shot voice cloning > Text + audio input/ Audio output > Ultra-fast inference w/ 200ms TTFA > Models on the Hub & Finetuning code on its way! 🚀 What an amazing time to be alive 🤗

Vaibhav (VB) Srivastav

66,979 Aufrufe • vor 1 Jahr

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.
4:10

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

AK

290,517 Aufrufe • vor 3 Jahren

We present MusicGen: A simple and controllable music generation model. MusicGen can be prompted by both text and melody. We release code (MIT) and models (CC-BY NC) for open research, reproducibility, and for the music community:
0:48

We present MusicGen: A simple and controllable music generation model. MusicGen can be prompted by both text and melody. We release code (MIT) and models (CC-BY NC) for open research, reproducibility, and for the music community:

Felix Kreuk

697,308 Aufrufe • vor 3 Jahren

(1/4) Excited to share our #ICCV2023 paper Text2Room! We generate scene-scale textured 3D meshes from a given text prompt leveraging 2D text-to-image models such as StableDiffusion. Project: Code: Video:
0:24

(1/4) Excited to share our #ICCV2023 paper Text2Room! We generate scene-scale textured 3D meshes from a given text prompt leveraging 2D text-to-image models such as StableDiffusion. Project: Code: Video:

Matthias Niessner

74,907 Aufrufe • vor 3 Jahren

Researchers at Meta recently shared MAGNeT, a single non-autoregressive transformer model for text-to-music & text-to-sound generation capable of generating audio on-par with the quality of SOTA models — at 7x the speed. MAGNeT is open source as part of AudioCraft. Hear audio samples and get more details on this work ➡️
0:51

Researchers at Meta recently shared MAGNeT, a single non-autoregressive transformer model for text-to-music & text-to-sound generation capable of generating audio on-par with the quality of SOTA models — at 7x the speed. MAGNeT is open source as part of AudioCraft. Hear audio samples and get more details on this work ➡️

AI at Meta

122,001 Aufrufe • vor 2 Jahren

WOW! DeepMind *just* dropped Magenta Real-time - Apache 2.0 licensed 🔥 > 800M params transformer, trained on ~190K hours of instrumental stock music > adapts MusicLM for real-time generation via 2s audio chunks (conditioned on prior 10s context) > 48 KHz Stereo > MusicCoCa: New joint music-text embedding model, blending MuLan and CoCa approaches > 1.25s generation time for 2s audio on free-tier Colab TPUs > style embeddings (from text/audio prompts) allow real-time morphing of genres/instruments > on-device inference, personal fine-tuning - coming soon! > model weights on Hugging Face 🤗
1:02

WOW! DeepMind *just* dropped Magenta Real-time - Apache 2.0 licensed 🔥 > 800M params transformer, trained on ~190K hours of instrumental stock music > adapts MusicLM for real-time generation via 2s audio chunks (conditioned on prior 10s context) > 48 KHz Stereo > MusicCoCa: New joint music-text embedding model, blending MuLan and CoCa approaches > 1.25s generation time for 2s audio on free-tier Colab TPUs > style embeddings (from text/audio prompts) allow real-time morphing of genres/instruments > on-device inference, personal fine-tuning - coming soon! > model weights on Hugging Face 🤗

Vaibhav (VB) Srivastav

90,219 Aufrufe • vor 1 Jahr

Oh wow! "[SIGGRAPH '24] DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models" Paper: Project: Code (MIT) :
1:04

Oh wow! "[SIGGRAPH '24] DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Models" Paper: Project: Code (MIT) :

MrNeRF

59,766 Aufrufe • vor 2 Jahren

MusicHiFi Fast High-Fidelity Stereo Vocoding Diffusion-based audio and music generation models commonly generate music by constructing an image representation of audio (e.g., a mel-spectrogram) and then converting it to audio using a phase reconstruction model or vocoder.
1:36

MusicHiFi Fast High-Fidelity Stereo Vocoding Diffusion-based audio and music generation models commonly generate music by constructing an image representation of audio (e.g., a mel-spectrogram) and then converting it to audio using a phase reconstruction model or vocoder.

AK

27,285 Aufrufe • vor 2 Jahren

Day 1 of 3 MLX Releases: Introducing MLX-Audio 🚀🔥 A text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple's MLX framework, providing efficient speech synthesis on Apple Silicon. Features ⚡️Fast inference on Apple Silicon (M series chips) 🤖Multiple language support 🗣️Voice customization options 🚀Quantization support for optimized performance Supported models: 🪶Kokoro - A multilingual TTS model with 82M params that supports various languages and voice styles. With more models coming soon. Get started: > pip install mlx-audio Please leave us a star and send a PR :)
4:07

Day 1 of 3 MLX Releases: Introducing MLX-Audio 🚀🔥 A text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple's MLX framework, providing efficient speech synthesis on Apple Silicon. Features ⚡️Fast inference on Apple Silicon (M series chips) 🤖Multiple language support 🗣️Voice customization options 🚀Quantization support for optimized performance Supported models: 🪶Kokoro - A multilingual TTS model with 82M params that supports various languages and voice styles. With more models coming soon. Get started: > pip install mlx-audio Please leave us a star and send a PR :)

Prince Canuma

123,544 Aufrufe • vor 1 Jahr

Mini-Omni 2 understands image, audio and text inputs all via end-to-end voice conversations with users 🔥 > Understands and processes images, speech, and text > Generates real-time speech responses > Supports interruptions during speech Technical Overview: > Concatenates image, audio, and text features for input. > Uses text-guided delayed parallel output for real-time speech > Involves encoder adaptation, modal alignment, and multimodal fine-tuning Best part: MIT licensed ⚡
0:54

Mini-Omni 2 understands image, audio and text inputs all via end-to-end voice conversations with users 🔥 > Understands and processes images, speech, and text > Generates real-time speech responses > Supports interruptions during speech Technical Overview: > Concatenates image, audio, and text features for input. > Uses text-guided delayed parallel output for real-time speech > Involves encoder adaptation, modal alignment, and multimodal fine-tuning Best part: MIT licensed ⚡

Vaibhav (VB) Srivastav

45,452 Aufrufe • vor 1 Jahr

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏
0:28

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏

Lior Alexander

231,624 Aufrufe • vor 3 Jahren

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.
2:43

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

AK

429,355 Aufrufe • vor 3 Jahren

🔥 DreamEngine revolutionizes image generation with its text-guided object fusion capabilities! The demo and code for Text Guided Object Fustion are released! Let's unlock the Imaginations! Run it locally now in: Paper:
0:27

🔥 DreamEngine revolutionizes image generation with its text-guided object fusion capabilities! The demo and code for Text Guided Object Fustion are released! Let's unlock the Imaginations! Run it locally now in: Paper:

Liang Chen

11,376 Aufrufe • vor 1 Jahr

📢📢📢 Excited to share our new work *Autonomous Character-Scene Interaction Synthesis from Text Instruction* (Siggraph Asia 24). It presents a unified model for flexible scene-conditioned motion generation given text, scene, trajectory conditions. The results with smooth interaction look very impressive! 📰Paper: Project: Code and data will be released soon.
6:34

📢📢📢 Excited to share our new work *Autonomous Character-Scene Interaction Synthesis from Text Instruction* (Siggraph Asia 24). It presents a unified model for flexible scene-conditioned motion generation given text, scene, trajectory conditions. The results with smooth interaction look very impressive! 📰Paper: Project: Code and data will be released soon.

Siyuan Huang

11,340 Aufrufe • vor 1 Jahr

We released Sonic-3.5 and Ink-2, the #1 streaming models for text to speech and speech to text you can use in your voice agents today. New architectures enable new frontiers for speed and quality. We're now the only provider to have #1 models for both speaking and listening.
2:47

We released Sonic-3.5 and Ink-2, the #1 streaming models for text to speech and speech to text you can use in your voice agents today. New architectures enable new frontiers for speed and quality. We're now the only provider to have #1 models for both speaking and listening.

Karan Goel

7,014,410 Aufrufe • vor 1 Monat

🗣️ 00:00 Intro 01:32 Audio agents 03:27 Speech-to-text 06:18 Text-to-speech 08:48 Agents SDK Read more in our blog post:
15:09

🗣️ 00:00 Intro 01:32 Audio agents 03:27 Speech-to-text 06:18 Text-to-speech 08:48 Agents SDK Read more in our blog post:

OpenAI Developers

494,145 Aufrufe • vor 1 Jahr