Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that... can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.show more

AK

481,358 subscribers

290,517 Aufrufe • vor 3 Jahren •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

9 Kommentare

Profilbild von Luyanda

Luyandavor 3 Jahren

Cool. When API?

Profilbild von Brandon

Brandonvor 3 Jahren

@yacineMTB I assume you’ve already seen this?

Profilbild von GibsonHimself

GibsonHimselfvor 3 Jahren

Google! STOP! GOOGLE! GOOGLE! Stop. I hope they're not allowing their models to learn on people's voice inputs. (Because they obviously are) Creepy AF

Profilbild von 𝘓𝘶𝘪𝘴 𝘎𝘰𝘮𝘦𝘻-𝘎𝘶𝘻𝘮𝘢𝘯

𝘓𝘶𝘪𝘴 𝘎𝘰𝘮𝘦𝘻-𝘎𝘶𝘻𝘮𝘢𝘯vor 3 Jahren

Great!.. no API...

Profilbild von jagarcia1980@gmail.com

[email protected]vor 3 Jahren

Se ve que con el español tenían algún tipo de problema o sesgo pero tranquilos, que casi nadie lo habla!.

Profilbild von act697

act697vor 3 Jahren

Show me Tamil sundar Pichai s mother tongue and then….

Profilbild von Jussac

Jussacvor 3 Jahren

Jenis suaranya juga bisa menyesuaikan dengan suara aslinya...

Profilbild von Chiara Maria Cervetta

Chiara Maria Cervettavor 3 Jahren

@_arohan_ This is very awesome! 😍 🤩 🌟

Profilbild von Damdi

Damdivor 3 Jahren

Google finally back to the game 😎

Ähnliche Videos

Introducing SeamlessM4T, the first all-in-one, multilingual multimodal translation model. This single model can perform tasks across speech-to-text, speech-to-speech, text-to-text translation & speech recognition for up to 100 languages depending on the task. Details ⬇️

Introducing SeamlessM4T, the first all-in-one, multilingual multimodal translation model. This single model can perform tasks across speech-to-text, speech-to-speech, text-to-text translation & speech recognition for up to 100 languages depending on the task. Details ⬇️

AI at Meta

592,726 Aufrufe • vor 2 Jahren

PaLM + AudioLM = AudioPaLM ! We start from PaLM pretrained on text and extend its vocab w/ audio tokens. This model can then be finetuned on a mix of any (speech, text) task e.g. ASR, TTS, MT and speech2speech translation in one's voice! 🧵1/4

PaLM + AudioLM = AudioPaLM ! We start from PaLM pretrained on text and extend its vocab w/ audio tokens. This model can then be finetuned on a mix of any (speech, text) task e.g. ASR, TTS, MT and speech2speech translation in one's voice! 🧵1/4

Neil Zeghidour

41,240 Aufrufe • vor 3 Jahren

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

AI at Meta

351,698 Aufrufe • vor 1 Jahr

In this demo, we show real-time voice translation between Indian languages. The user can speak in their preferred language, and the system translates and responds in another, preserving both meaning and natural delivery. Speech recognition, translation, and expressive text-to-speech work together seamlessly.

In this demo, we show real-time voice translation between Indian languages. The user can speak in their preferred language, and the system translates and responds in another, preserving both meaning and natural delivery. Speech recognition, translation, and expressive text-to-speech work together seamlessly.

Pratyush Kumar

15,206 Aufrufe • vor 4 Monaten

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

AK

429,294 Aufrufe • vor 3 Jahren

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,319 Aufrufe • vor 3 Jahren

MIT CSAIL and MyShell.AI researchers introduce OpenVoice V2, a text-to-speech model that can clone any voice and speak in many languages. Imagine your voice going global in multiple languages. OpenVoice V2 breaks the language barrier and redefines voice interactions.

MIT CSAIL and MyShell.AI researchers introduce OpenVoice V2, a text-to-speech model that can clone any voice and speak in many languages. Imagine your voice going global in multiple languages. OpenVoice V2 breaks the language barrier and redefines voice interactions.

MIT CSAIL

66,460 Aufrufe • vor 2 Jahren

Introduce OpenVoice V2 - a Text-to-Speech model that can clone any voice and speak in any language. Developed by MyShell and MIT CSAIL researchers. 🌐 Imagine your voice going global in multiple languages. 🔊 OpenVoice V2 breaks the language barrier and redefines voice interaction.

Introduce OpenVoice V2 - a Text-to-Speech model that can clone any voice and speak in any language. Developed by MyShell and MIT CSAIL researchers. 🌐 Imagine your voice going global in multiple languages. 🔊 OpenVoice V2 breaks the language barrier and redefines voice interaction.

MyShell.AI

292,515 Aufrufe • vor 2 Jahren

Meet Hibiki, our simultaneous speech-to-speech translation model, currently supporting 🇫🇷➡️🇬🇧. Hibiki produces spoken and text translations of the input speech in real-time, while preserving the speaker’s voice and optimally adapting its pace based on the semantic content of the source speech. Based on objective and human evaluations, Hibiki outperforms previous systems for quality, naturalness and speaker similarity and approaches human interpreters. 🧵

Meet Hibiki, our simultaneous speech-to-speech translation model, currently supporting 🇫🇷➡️🇬🇧. Hibiki produces spoken and text translations of the input speech in real-time, while preserving the speaker’s voice and optimally adapting its pace based on the semantic content of the source speech. Based on objective and human evaluations, Hibiki outperforms previous systems for quality, naturalness and speaker similarity and approaches human interpreters. 🧵

kyutai

167,428 Aufrufe • vor 1 Jahr

I made a few updates to the MLX port of F5 TTS over the holiday: - Longform generation is now supported, and text will be split automatically on sentence boundaries for natural-sounding speech. - You can now pipe in text from another process (e.g. a language model) and listen to the speech as each segment is generated. - I implemented RK4 sampling for the ODE, which achieves much better quality with fewer steps — it's now possible to generate speech at 1.6x realtime on my M3 Max. See the video of an example of generating speech in realtime from a language model entirely on-device! 🚀

I made a few updates to the MLX port of F5 TTS over the holiday: - Longform generation is now supported, and text will be split automatically on sentence boundaries for natural-sounding speech. - You can now pipe in text from another process (e.g. a language model) and listen to the speech as each segment is generated. - I implemented RK4 sampling for the ODE, which achieves much better quality with fewer steps — it's now possible to generate speech at 1.6x realtime on my M3 Max. See the video of an example of generating speech in realtime from a language model entirely on-device! 🚀

Lucas Newman

22,962 Aufrufe • vor 1 Jahr

We have officially launched Fun-ASR1.5, a major update to our end-to-end speech recognition model. This release focuses on three core pillars: broader language coverage, language switching, and production-ready text output. Key Features: • Multilingual Support: Supports high-accuracy recognition for 30 languages across Asia, Europe, and the Middle East within a single model. • Language Switching: Handles mixed-language speech (Code-Switching) natively, automatically detecting and transcribing language shifts without the need for manual tagging. • Professional Text Output: Delivers "ready-to-use" text with smart punctuation and automatic formatting for dates, numbers, and currencies. Fun-ASR1.5 bridges the gap between raw audio and professional documentation, providing a reliable engine for global communication.

We have officially launched Fun-ASR1.5, a major update to our end-to-end speech recognition model. This release focuses on three core pillars: broader language coverage, language switching, and production-ready text output. Key Features: • Multilingual Support: Supports high-accuracy recognition for 30 languages across Asia, Europe, and the Middle East within a single model. • Language Switching: Handles mixed-language speech (Code-Switching) natively, automatically detecting and transcribing language shifts without the need for manual tagging. • Professional Text Output: Delivers "ready-to-use" text with smart punctuation and automatic formatting for dates, numbers, and currencies. Fun-ASR1.5 bridges the gap between raw audio and professional documentation, providing a reliable engine for global communication.

Tongyi Lab

3,917,307 Aufrufe • vor 2 Monaten

Introducing Scribe — the most accurate Speech to Text model. It has the highest accuracy on benchmarks, outperforming previous state-of-the-art models such as Gemini 2.0 and OpenAI Whisper v3. It’s now the leading model for English, Spanish, Italian, and many more. With support for 99 languages, speaker diarization, character-level timestamps, and non-speech events such as laughing.

Introducing Scribe — the most accurate Speech to Text model. It has the highest accuracy on benchmarks, outperforming previous state-of-the-art models such as Gemini 2.0 and OpenAI Whisper v3. It’s now the leading model for English, Spanish, Italian, and many more. With support for 99 languages, speaker diarization, character-level timestamps, and non-speech events such as laughing.

ElevenLabs

464,489 Aufrufe • vor 1 Jahr

We released Sonic-3.5 and Ink-2, the #1 streaming models for text to speech and speech to text you can use in your voice agents today. New architectures enable new frontiers for speed and quality. We're now the only provider to have #1 models for both speaking and listening.

We released Sonic-3.5 and Ink-2, the #1 streaming models for text to speech and speech to text you can use in your voice agents today. New architectures enable new frontiers for speed and quality. We're now the only provider to have #1 models for both speaking and listening.

Karan Goel

7,003,631 Aufrufe • vor 25 Tagen

LP-MusicCaps: LLM-Based Pseudo Music Captioning paper page: Automatic music captioning, which generates natural language descriptions for given music tracks, holds significant potential for enhancing the understanding and organization of large volumes of musical data. Despite its importance, researchers face challenges due to the costly and time-consuming collection process of existing music-language datasets, which are limited in size. To address this data scarcity issue, we propose the use of large language models (LLMs) to artificially generate the description sentences from large-scale tag datasets. This results in approximately 2.2M captions paired with 0.5M audio clips. We term it Large Language Model based Pseudo music caption dataset, shortly, LP-MusicCaps. We conduct a systemic evaluation of the large-scale music captioning dataset with various quantitative evaluation metrics used in the field of natural language processing as well as human evaluation. In addition, we trained a transformer-based music captioning model with the dataset and evaluated it under zero-shot and transfer-learning settings. The results demonstrate that our proposed approach outperforms the supervised baseline model.

LP-MusicCaps: LLM-Based Pseudo Music Captioning paper page: Automatic music captioning, which generates natural language descriptions for given music tracks, holds significant potential for enhancing the understanding and organization of large volumes of musical data. Despite its importance, researchers face challenges due to the costly and time-consuming collection process of existing music-language datasets, which are limited in size. To address this data scarcity issue, we propose the use of large language models (LLMs) to artificially generate the description sentences from large-scale tag datasets. This results in approximately 2.2M captions paired with 0.5M audio clips. We term it Large Language Model based Pseudo music caption dataset, shortly, LP-MusicCaps. We conduct a systemic evaluation of the large-scale music captioning dataset with various quantitative evaluation metrics used in the field of natural language processing as well as human evaluation. In addition, we trained a transformer-based music captioning model with the dataset and evaluated it under zero-shot and transfer-learning settings. The results demonstrate that our proposed approach outperforms the supervised baseline model.

AK

78,794 Aufrufe • vor 2 Jahren

Stream-Omni Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

Stream-Omni Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

AK

12,374 Aufrufe • vor 1 Jahr

Today we launched Gemini 3.1 Flash TTS, our most expressive and controllable text-to-speech model yet. This launch [excitement] includes audio tags! 🗣🏷 Audio tags [explanatory] are a seamless way to guide vocal style, pace, and delivery using natural language commands embedded directly in your text. Want a different tempo or tone? [amazement] Just tag the audio to steer the AI-speech output! The model supports 70+ languages (24 of which are high-quality evaluated languages, including: Japanese, Hindi, and Arabic). Watch the audio tags in action in the demo below ↓

Today we launched Gemini 3.1 Flash TTS, our most expressive and controllable text-to-speech model yet. This launch [excitement] includes audio tags! 🗣🏷 Audio tags [explanatory] are a seamless way to guide vocal style, pace, and delivery using natural language commands embedded directly in your text. Want a different tempo or tone? [amazement] Just tag the audio to steer the AI-speech output! The model supports 70+ languages (24 of which are high-quality evaluated languages, including: Japanese, Hindi, and Arabic). Watch the audio tags in action in the demo below ↓

Google AI

202,515 Aufrufe • vor 2 Monaten

Listen up 🔊 We’ve made some updates to our Gemini Audio models and capabilities: — Gemini’s live speech-to-speech translation capability is rolling out in a beta experience to the Google Translate app, bringing you real-time audio translation that captures the nuance of human speech — Gemini 2.5 Flash and 2.5 Pro Text-to-Speech preview models bring improved adherence to style prompts, precision pacing with context-aware speed adjustments, and character voice consistency for multi-speaker scenarios — Gemini 2.5 Flash Native Audio is now updated, with improvements to handle complex workflows, navigate user instructions, and hold natural conversations

Listen up 🔊 We’ve made some updates to our Gemini Audio models and capabilities: — Gemini’s live speech-to-speech translation capability is rolling out in a beta experience to the Google Translate app, bringing you real-time audio translation that captures the nuance of human speech — Gemini 2.5 Flash and 2.5 Pro Text-to-Speech preview models bring improved adherence to style prompts, precision pacing with context-aware speed adjustments, and character voice consistency for multi-speaker scenarios — Gemini 2.5 Flash Native Audio is now updated, with improvements to handle complex workflows, navigate user instructions, and hold natural conversations

Google AI

1,092,358 Aufrufe • vor 7 Monaten

Day 1 of 3 MLX Releases: Introducing MLX-Audio 🚀🔥 A text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple's MLX framework, providing efficient speech synthesis on Apple Silicon. Features ⚡️Fast inference on Apple Silicon (M series chips) 🤖Multiple language support 🗣️Voice customization options 🚀Quantization support for optimized performance Supported models: 🪶Kokoro - A multilingual TTS model with 82M params that supports various languages and voice styles. With more models coming soon. Get started: > pip install mlx-audio Please leave us a star and send a PR :)

Day 1 of 3 MLX Releases: Introducing MLX-Audio 🚀🔥 A text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple's MLX framework, providing efficient speech synthesis on Apple Silicon. Features ⚡️Fast inference on Apple Silicon (M series chips) 🤖Multiple language support 🗣️Voice customization options 🚀Quantization support for optimized performance Supported models: 🪶Kokoro - A multilingual TTS model with 82M params that supports various languages and voice styles. With more models coming soon. Get started: > pip install mlx-audio Please leave us a star and send a PR :)

Prince Canuma

123,480 Aufrufe • vor 1 Jahr

We pioneered the first ultra-realistic Text to Speech model, and recently launched the world's most accurate Speech to Text model, Scribe. But we're not stopping there. Today, we're taking one small step for man, and one giant leap for man's best friend... with Text to Bark.

We pioneered the first ultra-realistic Text to Speech model, and recently launched the world's most accurate Speech to Text model, Scribe. But we're not stopping there. Today, we're taking one small step for man, and one giant leap for man's best friend... with Text to Bark.

ElevenLabs

291,233 Aufrufe • vor 1 Jahr

The new open-source Text to Speech model: Fish Speech 1.4 is brilliant! Trained on a massive 700K hours of multilingual speech data in 8 languages - Instant voice cloning 🗣️ - Ultra-low latency ⚡ - Compact model (~1GB weights) 🏋️‍♂️

The new open-source Text to Speech model: Fish Speech 1.4 is brilliant! Trained on a massive 700K hours of multilingual speech data in 8 languages - Instant voice cloning 🗣️ - Ultra-low latency ⚡ - Compact model (~1GB weights) 🏋️‍♂️

Rohan Paul

228,836 Aufrufe • vor 1 Jahr