Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that... can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.show more

AK

481,358 subscribers

290,517 görüntüleme • 3 yıl önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

9 Yorum

Luyanda profil fotoğrafı

Luyanda3 yıl önce

Cool. When API?

Brandon profil fotoğrafı

Brandon3 yıl önce

@yacineMTB I assume you’ve already seen this?

GibsonHimself profil fotoğrafı

GibsonHimself3 yıl önce

Google! STOP! GOOGLE! GOOGLE! Stop. I hope they're not allowing their models to learn on people's voice inputs. (Because they obviously are) Creepy AF

𝘓𝘶𝘪𝘴 𝘎𝘰𝘮𝘦𝘻-𝘎𝘶𝘻𝘮𝘢𝘯 profil fotoğrafı

𝘓𝘶𝘪𝘴 𝘎𝘰𝘮𝘦𝘻-𝘎𝘶𝘻𝘮𝘢𝘯3 yıl önce

Great!.. no API...

jagarcia1980@gmail.com profil fotoğrafı

[email protected]3 yıl önce

Se ve que con el español tenían algún tipo de problema o sesgo pero tranquilos, que casi nadie lo habla!.

act697 profil fotoğrafı

act6973 yıl önce

Show me Tamil sundar Pichai s mother tongue and then….

Jussac profil fotoğrafı

Jussac3 yıl önce

Jenis suaranya juga bisa menyesuaikan dengan suara aslinya...

Chiara Maria Cervetta profil fotoğrafı

Chiara Maria Cervetta3 yıl önce

@_arohan_ This is very awesome! 😍 🤩 🌟

Damdi profil fotoğrafı

Damdi3 yıl önce

Google finally back to the game 😎

Benzer Videolar

PaLM + AudioLM = AudioPaLM ! We start from PaLM pretrained on text and extend its vocab w/ audio tokens. This model can then be finetuned on a mix of any (speech, text) task e.g. ASR, TTS, MT and speech2speech translation in one's voice! 🧵1/4

PaLM + AudioLM = AudioPaLM ! We start from PaLM pretrained on text and extend its vocab w/ audio tokens. This model can then be finetuned on a mix of any (speech, text) task e.g. ASR, TTS, MT and speech2speech translation in one's voice! 🧵1/4

Neil Zeghidour

41,240 görüntüleme • 3 yıl önce

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

AI at Meta

351,764 görüntüleme • 1 yıl önce

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale blog: Large-scale generative models such as GPT and DALL-E have revolutionized natural language processing and computer vision research. These models not only generate high fidelity text or image outputs, but are also generalists which can solve tasks not explicitly taught. In contrast, speech generative models are still primitive in terms of scale and task generalization. In this paper, we present Voicebox, the most versatile text-guided generative model for speech at scale. Voicebox is a non-autoregressive flow-matching model trained to infill speech, given audio context and text, trained on over 50K hours of speech that are neither filtered nor enhanced. Similar to GPT, Voicebox can perform many different tasks through in-context learning, but is more flexible as it can also condition on future context. Voicebox can be used for mono or cross-lingual zero-shot text-to-speech synthesis, noise removal, content editing, style conversion, and diverse sample generation. In particular, Voicebox outperforms the state-of-the-art zero-shot TTS model VALL-E on both intelligibility (5.9% vs 1.9% word error rates) and audio similarity (0.580 vs 0.681) while being up to 20 times faster.

AK

429,355 görüntüleme • 3 yıl önce

In this demo, we show real-time voice translation between Indian languages. The user can speak in their preferred language, and the system translates and responds in another, preserving both meaning and natural delivery. Speech recognition, translation, and expressive text-to-speech work together seamlessly.

In this demo, we show real-time voice translation between Indian languages. The user can speak in their preferred language, and the system translates and responds in another, preserving both meaning and natural delivery. Speech recognition, translation, and expressive text-to-speech work together seamlessly.

Pratyush Kumar

15,206 görüntüleme • 6 ay önce

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,353 görüntüleme • 3 yıl önce

MIT CSAIL and MyShell.AI researchers introduce OpenVoice V2, a text-to-speech model that can clone any voice and speak in many languages. Imagine your voice going global in multiple languages. OpenVoice V2 breaks the language barrier and redefines voice interactions.

MIT CSAIL and MyShell.AI researchers introduce OpenVoice V2, a text-to-speech model that can clone any voice and speak in many languages. Imagine your voice going global in multiple languages. OpenVoice V2 breaks the language barrier and redefines voice interactions.

MIT CSAIL

66,460 görüntüleme • 2 yıl önce

Meet Hibiki, our simultaneous speech-to-speech translation model, currently supporting 🇫🇷➡️🇬🇧. Hibiki produces spoken and text translations of the input speech in real-time, while preserving the speaker’s voice and optimally adapting its pace based on the semantic content of the source speech. Based on objective and human evaluations, Hibiki outperforms previous systems for quality, naturalness and speaker similarity and approaches human interpreters. 🧵

Meet Hibiki, our simultaneous speech-to-speech translation model, currently supporting 🇫🇷➡️🇬🇧. Hibiki produces spoken and text translations of the input speech in real-time, while preserving the speaker’s voice and optimally adapting its pace based on the semantic content of the source speech. Based on objective and human evaluations, Hibiki outperforms previous systems for quality, naturalness and speaker similarity and approaches human interpreters. 🧵

kyutai

167,614 görüntüleme • 1 yıl önce

Introduce OpenVoice V2 - a Text-to-Speech model that can clone any voice and speak in any language. Developed by MyShell and MIT CSAIL researchers. 🌐 Imagine your voice going global in multiple languages. 🔊 OpenVoice V2 breaks the language barrier and redefines voice interaction.

Introduce OpenVoice V2 - a Text-to-Speech model that can clone any voice and speak in any language. Developed by MyShell and MIT CSAIL researchers. 🌐 Imagine your voice going global in multiple languages. 🔊 OpenVoice V2 breaks the language barrier and redefines voice interaction.

MyShell.AI

292,515 görüntüleme • 2 yıl önce

I made a few updates to the MLX port of F5 TTS over the holiday: - Longform generation is now supported, and text will be split automatically on sentence boundaries for natural-sounding speech. - You can now pipe in text from another process (e.g. a language model) and listen to the speech as each segment is generated. - I implemented RK4 sampling for the ODE, which achieves much better quality with fewer steps — it's now possible to generate speech at 1.6x realtime on my M3 Max. See the video of an example of generating speech in realtime from a language model entirely on-device! 🚀

I made a few updates to the MLX port of F5 TTS over the holiday: - Longform generation is now supported, and text will be split automatically on sentence boundaries for natural-sounding speech. - You can now pipe in text from another process (e.g. a language model) and listen to the speech as each segment is generated. - I implemented RK4 sampling for the ODE, which achieves much better quality with fewer steps — it's now possible to generate speech at 1.6x realtime on my M3 Max. See the video of an example of generating speech in realtime from a language model entirely on-device! 🚀

Lucas Newman

22,998 görüntüleme • 1 yıl önce

We have officially launched Fun-ASR1.5, a major update to our end-to-end speech recognition model. This release focuses on three core pillars: broader language coverage, language switching, and production-ready text output. Key Features: • Multilingual Support: Supports high-accuracy recognition for 30 languages across Asia, Europe, and the Middle East within a single model. • Language Switching: Handles mixed-language speech (Code-Switching) natively, automatically detecting and transcribing language shifts without the need for manual tagging. • Professional Text Output: Delivers "ready-to-use" text with smart punctuation and automatic formatting for dates, numbers, and currencies. Fun-ASR1.5 bridges the gap between raw audio and professional documentation, providing a reliable engine for global communication.

We have officially launched Fun-ASR1.5, a major update to our end-to-end speech recognition model. This release focuses on three core pillars: broader language coverage, language switching, and production-ready text output. Key Features: • Multilingual Support: Supports high-accuracy recognition for 30 languages across Asia, Europe, and the Middle East within a single model. • Language Switching: Handles mixed-language speech (Code-Switching) natively, automatically detecting and transcribing language shifts without the need for manual tagging. • Professional Text Output: Delivers "ready-to-use" text with smart punctuation and automatic formatting for dates, numbers, and currencies. Fun-ASR1.5 bridges the gap between raw audio and professional documentation, providing a reliable engine for global communication.

Tongyi Lab

3,917,307 görüntüleme • 4 ay önce

Introducing Scribe — the most accurate Speech to Text model. It has the highest accuracy on benchmarks, outperforming previous state-of-the-art models such as Gemini 2.0 and OpenAI Whisper v3. It’s now the leading model for English, Spanish, Italian, and many more. With support for 99 languages, speaker diarization, character-level timestamps, and non-speech events such as laughing.

Introducing Scribe — the most accurate Speech to Text model. It has the highest accuracy on benchmarks, outperforming previous state-of-the-art models such as Gemini 2.0 and OpenAI Whisper v3. It’s now the leading model for English, Spanish, Italian, and many more. With support for 99 languages, speaker diarization, character-level timestamps, and non-speech events such as laughing.

ElevenLabs

464,836 görüntüleme • 1 yıl önce

🎙️ Dictate in Multiple Languages in #VSCode Dictation can detect your language and turn your speech into text using a local model. You can use it in Chat, the editor, and the integrated terminal.

🎙️ Dictate in Multiple Languages in #VSCode Dictation can detect your language and turn your speech into text using a local model. You can use it in Chat, the editor, and the integrated terminal.

Visual Studio Code

27,733 görüntüleme • 12 gün önce

LP-MusicCaps: LLM-Based Pseudo Music Captioning paper page: Automatic music captioning, which generates natural language descriptions for given music tracks, holds significant potential for enhancing the understanding and organization of large volumes of musical data. Despite its importance, researchers face challenges due to the costly and time-consuming collection process of existing music-language datasets, which are limited in size. To address this data scarcity issue, we propose the use of large language models (LLMs) to artificially generate the description sentences from large-scale tag datasets. This results in approximately 2.2M captions paired with 0.5M audio clips. We term it Large Language Model based Pseudo music caption dataset, shortly, LP-MusicCaps. We conduct a systemic evaluation of the large-scale music captioning dataset with various quantitative evaluation metrics used in the field of natural language processing as well as human evaluation. In addition, we trained a transformer-based music captioning model with the dataset and evaluated it under zero-shot and transfer-learning settings. The results demonstrate that our proposed approach outperforms the supervised baseline model.

LP-MusicCaps: LLM-Based Pseudo Music Captioning paper page: Automatic music captioning, which generates natural language descriptions for given music tracks, holds significant potential for enhancing the understanding and organization of large volumes of musical data. Despite its importance, researchers face challenges due to the costly and time-consuming collection process of existing music-language datasets, which are limited in size. To address this data scarcity issue, we propose the use of large language models (LLMs) to artificially generate the description sentences from large-scale tag datasets. This results in approximately 2.2M captions paired with 0.5M audio clips. We term it Large Language Model based Pseudo music caption dataset, shortly, LP-MusicCaps. We conduct a systemic evaluation of the large-scale music captioning dataset with various quantitative evaluation metrics used in the field of natural language processing as well as human evaluation. In addition, we trained a transformer-based music captioning model with the dataset and evaluated it under zero-shot and transfer-learning settings. The results demonstrate that our proposed approach outperforms the supervised baseline model.

AK

78,794 görüntüleme • 3 yıl önce

Today we launched Gemini 3.1 Flash TTS, our most expressive and controllable text-to-speech model yet. This launch [excitement] includes audio tags! 🗣🏷 Audio tags [explanatory] are a seamless way to guide vocal style, pace, and delivery using natural language commands embedded directly in your text. Want a different tempo or tone? [amazement] Just tag the audio to steer the AI-speech output! The model supports 70+ languages (24 of which are high-quality evaluated languages, including: Japanese, Hindi, and Arabic). Watch the audio tags in action in the demo below ↓

Today we launched Gemini 3.1 Flash TTS, our most expressive and controllable text-to-speech model yet. This launch [excitement] includes audio tags! 🗣🏷 Audio tags [explanatory] are a seamless way to guide vocal style, pace, and delivery using natural language commands embedded directly in your text. Want a different tempo or tone? [amazement] Just tag the audio to steer the AI-speech output! The model supports 70+ languages (24 of which are high-quality evaluated languages, including: Japanese, Hindi, and Arabic). Watch the audio tags in action in the demo below ↓

Google AI

203,166 görüntüleme • 4 ay önce

Day 1 of 3 MLX Releases: Introducing MLX-Audio 🚀🔥 A text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple's MLX framework, providing efficient speech synthesis on Apple Silicon. Features ⚡️Fast inference on Apple Silicon (M series chips) 🤖Multiple language support 🗣️Voice customization options 🚀Quantization support for optimized performance Supported models: 🪶Kokoro - A multilingual TTS model with 82M params that supports various languages and voice styles. With more models coming soon. Get started: > pip install mlx-audio Please leave us a star and send a PR :)

Day 1 of 3 MLX Releases: Introducing MLX-Audio 🚀🔥 A text-to-speech (TTS) and Speech-to-Speech (STS) library built on Apple's MLX framework, providing efficient speech synthesis on Apple Silicon. Features ⚡️Fast inference on Apple Silicon (M series chips) 🤖Multiple language support 🗣️Voice customization options 🚀Quantization support for optimized performance Supported models: 🪶Kokoro - A multilingual TTS model with 82M params that supports various languages and voice styles. With more models coming soon. Get started: > pip install mlx-audio Please leave us a star and send a PR :)

Prince Canuma

123,553 görüntüleme • 1 yıl önce

Listen up 🔊 We’ve made some updates to our Gemini Audio models and capabilities: — Gemini’s live speech-to-speech translation capability is rolling out in a beta experience to the Google Translate app, bringing you real-time audio translation that captures the nuance of human speech — Gemini 2.5 Flash and 2.5 Pro Text-to-Speech preview models bring improved adherence to style prompts, precision pacing with context-aware speed adjustments, and character voice consistency for multi-speaker scenarios — Gemini 2.5 Flash Native Audio is now updated, with improvements to handle complex workflows, navigate user instructions, and hold natural conversations

Listen up 🔊 We’ve made some updates to our Gemini Audio models and capabilities: — Gemini’s live speech-to-speech translation capability is rolling out in a beta experience to the Google Translate app, bringing you real-time audio translation that captures the nuance of human speech — Gemini 2.5 Flash and 2.5 Pro Text-to-Speech preview models bring improved adherence to style prompts, precision pacing with context-aware speed adjustments, and character voice consistency for multi-speaker scenarios — Gemini 2.5 Flash Native Audio is now updated, with improvements to handle complex workflows, navigate user instructions, and hold natural conversations

Google AI

1,093,819 görüntüleme • 8 ay önce

OpenAI's S2S preview is polished but it still thinks in steps. Speech → text → model → text → speech. That's not how humans converse. Introducing Hydra. A native speech-to-speech model that doesn't wait for turn-taking, doesn't flatten emotion into text, and doesn't break when you interrupt it mid-sentence. Hydra reasons asynchronously, speaks and listens simultaneously, and preserves emotion because it never leaves the audio domain. It's still in beta, but the shift is obvious. If you want early access, the link is in the comments. Here's a preview of what that looks like -

OpenAI's S2S preview is polished but it still thinks in steps. Speech → text → model → text → speech. That's not how humans converse. Introducing Hydra. A native speech-to-speech model that doesn't wait for turn-taking, doesn't flatten emotion into text, and doesn't break when you interrupt it mid-sentence. Hydra reasons asynchronously, speaks and listens simultaneously, and preserves emotion because it never leaves the audio domain. It's still in beta, but the shift is obvious. If you want early access, the link is in the comments. Here's a preview of what that looks like -

Sudarshan Kamath

329,032 görüntüleme • 5 ay önce

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,657 görüntüleme • 3 yıl önce

Eleven v3 (alpha) is the most expressive Text to Speech model. v3 introduces: • Multi-speaker dialogue with contextual awareness • Support for 70+ languages, up from 33 in v2 • Audio tags such as [excited], [sighs], [laughing], and [whispers]

Eleven v3 (alpha) is the most expressive Text to Speech model. v3 introduces: • Multi-speaker dialogue with contextual awareness • Support for 70+ languages, up from 33 in v2 • Audio tags such as [excited], [sighs], [laughing], and [whispers]

ElevenLabs

39,424 görüntüleme • 1 yıl önce