正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

does your audio-input LLM secretly know how to generate speech? yes but it sounds like a scary demon! deepdream but for speech: starting with noise audio, I tried optimizing the probability of a desired transcription, while keeping the model frozen. volume up! 🎧

Matt Henderson

81,435 subscribers

86,530 次观看 • 1 个月前 •via X (Twitter)

科学技术教育音乐

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Along with text, images, video and code, Gemini is able to process raw audio signal end-to-end. 🔊 It can listen to and understand speech, making it not only useful for transcription but a model that has a much more nuanced perception of its environment. ↓

Along with text, images, video and code, Gemini is able to process raw audio signal end-to-end. 🔊 It can listen to and understand speech, making it not only useful for transcription but a model that has a much more nuanced perception of its environment. ↓

Google DeepMind

140,179 次观看 • 2 年前

I love using transcription when taking notes, but i’m not a huge fan of a lot of the apps that offer it. I also don’t like in-meeting note takers. So naturally I built a Raycast extension for transcribing notes. It transcribes both audio input and system audio and uses AI to summarise and copy to clipboard

I love using transcription when taking notes, but i’m not a huge fan of a lot of the apps that offer it. I also don’t like in-meeting note takers. So naturally I built a Raycast extension for transcribing notes. It transcribes both audio input and system audio and uses AI to summarise and copy to clipboard

Thomas Cullen

13,965 次观看 • 4 个月前

OpenAI's S2S preview is polished but it still thinks in steps. Speech → text → model → text → speech. That's not how humans converse. Introducing Hydra. A native speech-to-speech model that doesn't wait for turn-taking, doesn't flatten emotion into text, and doesn't break when you interrupt it mid-sentence. Hydra reasons asynchronously, speaks and listens simultaneously, and preserves emotion because it never leaves the audio domain. It's still in beta, but the shift is obvious. If you want early access, the link is in the comments. Here's a preview of what that looks like -

OpenAI's S2S preview is polished but it still thinks in steps. Speech → text → model → text → speech. That's not how humans converse. Introducing Hydra. A native speech-to-speech model that doesn't wait for turn-taking, doesn't flatten emotion into text, and doesn't break when you interrupt it mid-sentence. Hydra reasons asynchronously, speaks and listens simultaneously, and preserves emotion because it never leaves the audio domain. It's still in beta, but the shift is obvious. If you want early access, the link is in the comments. Here's a preview of what that looks like -

Sudarshan Kamath

329,032 次观看 • 6 个月前

This is not a joke! 🐬 Excited to share DolphinGemma the first audio-to-audio for dolphin communication! Yes, a model that predicts tokens on how dolphin speech! > DolphinGemma is the first LLM trained specifically to understand dolphin language patterns. > Leverages 40 years of data from Dr. Denise Herzing's unique collection > Works like text prediction, trying to "complete" dolphin whistles and sounds > Use wearable hardware (Google Pixel 9) to capture and analyze sounds in the field. > Dolphin Gemma is designed to be fine-tuned with new data > Weights coming soon! Research like this is why I love AI even more! ♥️

This is not a joke! 🐬 Excited to share DolphinGemma the first audio-to-audio for dolphin communication! Yes, a model that predicts tokens on how dolphin speech! > DolphinGemma is the first LLM trained specifically to understand dolphin language patterns. > Leverages 40 years of data from Dr. Denise Herzing's unique collection > Works like text prediction, trying to "complete" dolphin whistles and sounds > Use wearable hardware (Google Pixel 9) to capture and analyze sounds in the field. > Dolphin Gemma is designed to be fine-tuned with new data > Weights coming soon! Research like this is why I love AI even more! ♥️

Philipp Schmid

112,244 次观看 • 1 年前

You can now generate real-time speech that sounds conversational. Microsoft just open-sourced VibeVoice, a real-time text-to-speech system with ~300 ms first audio latency and streaming input. It handles long conversations without falling apart. 𝗧𝗵𝗶𝘀 𝗺𝗼𝗱𝗲𝗹 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲𝘀 𝗹𝗼𝗻𝗴, 𝗺𝘂𝗹𝘁𝗶-𝘀𝗽𝗲𝗮𝗸𝗲𝗿 𝘀𝗽𝗲𝗲𝗰𝗵. It produces up to 90 minutes of audio. It supports up to four distinct speakers. Turn-taking stays consistent over long sessions. 𝗜𝘁 𝘄𝗼𝗿𝗸𝘀 𝗯𝘆 𝗿𝗲𝗱𝘂𝗰𝗶𝗻𝗴 𝘁𝗶𝗺𝗲 𝗿𝗲𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻. Audio compresses into semantic and acoustic tokens. They run at 7.5 Hz instead of frame-level audio. A language model predicts structure. A diffusion head restores acoustic detail. 𝗜𝘁 𝗮𝗹𝗹𝗼𝘄𝘀 𝗹𝗼𝘄-𝗹𝗮𝘁𝗲𝗻𝗰𝘆 𝘀𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴 𝗮𝘂𝗱𝗶𝗼. The real-time variant streams text incrementally. First speech arrives in ~300 ms. A WebSocket demo shows live generation. The code is MIT-licensed and research-only. The repo already passed 20k GitHub stars.

You can now generate real-time speech that sounds conversational. Microsoft just open-sourced VibeVoice, a real-time text-to-speech system with ~300 ms first audio latency and streaming input. It handles long conversations without falling apart. 𝗧𝗵𝗶𝘀 𝗺𝗼𝗱𝗲𝗹 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲𝘀 𝗹𝗼𝗻𝗴, 𝗺𝘂𝗹𝘁𝗶-𝘀𝗽𝗲𝗮𝗸𝗲𝗿 𝘀𝗽𝗲𝗲𝗰𝗵. It produces up to 90 minutes of audio. It supports up to four distinct speakers. Turn-taking stays consistent over long sessions. 𝗜𝘁 𝘄𝗼𝗿𝗸𝘀 𝗯𝘆 𝗿𝗲𝗱𝘂𝗰𝗶𝗻𝗴 𝘁𝗶𝗺𝗲 𝗿𝗲𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻. Audio compresses into semantic and acoustic tokens. They run at 7.5 Hz instead of frame-level audio. A language model predicts structure. A diffusion head restores acoustic detail. 𝗜𝘁 𝗮𝗹𝗹𝗼𝘄𝘀 𝗹𝗼𝘄-𝗹𝗮𝘁𝗲𝗻𝗰𝘆 𝘀𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴 𝗮𝘂𝗱𝗶𝗼. The real-time variant streams text incrementally. First speech arrives in ~300 ms. A WebSocket demo shows live generation. The code is MIT-licensed and research-only. The repo already passed 20k GitHub stars.

Lior Alexander

61,122 次观看 • 7 个月前

Audio Transcription with Google Gemini 1.5 Flash In this video, Gemini-Flash was able to transcribe 13 minutes of audio in 50-60 seconds. I have tested this with multiple audio files, and the transcription accuracy is close to 99%. If the audio is clear, you get 100% correct transcription. Even if the audio is really bad with a lot of noise, you still get around 95-96% accuracy. The code for this is available on GitHub. If you're interested, you can download and run it. You just need a Google API key.

Audio Transcription with Google Gemini 1.5 Flash In this video, Gemini-Flash was able to transcribe 13 minutes of audio in 50-60 seconds. I have tested this with multiple audio files, and the transcription accuracy is close to 99%. If the audio is clear, you get 100% correct transcription. Even if the audio is really bad with a lot of noise, you still get around 95-96% accuracy. The code for this is available on GitHub. If you're interested, you can download and run it. You just need a Google API key.

AshutoshShrivastava

122,990 次观看 • 1 年前

Today we’re launching Stable Audio 2.5: The first audio model built for enterprise-grade sound production 🔊 Audio influences brand engagement by 86%, but few enterprises are leveraging audio as an extension of their brand, making customized sound an untapped differentiator. Stable Audio 2.5 is purpose-built for this opportunity to create customizable, high-quality audio at scale, with capabilities that include: ▶️ Improved musical composition: Generate full songs with multi-part structure, meaning a clear intro, middle, and outro. ▶️ Audio inpainting: Input audio, select where the track should start, and the model uses the context to generate the rest of the track. ▶️ Customization: Our team can fine-tune Stable Audio 2.5 to help enterprises create the right sound for their brand. ▶️ Faster inference: The model can generate up to three-minute long tracks in under two seconds on a GPU, outputting in just eight steps (compared to ~50 in the previous model). You can learn more here 👉

Stability AI

68,171 次观看 • 1 年前

Elon Musk's Wisconsin Town Hall Q&A + Speech ✅ Improved audio for easier listening ✅ Silences removed to save you time Q&A at start, Elon's speech starts at 1h 25m. FYI, in this edit I put the Q&A *first* but at the event, the Q&A followed the speech. Separate edits are coming with just the speech (and) one with just the Q&A. Share them with someone you know!

Elon Musk's Wisconsin Town Hall Q&A + Speech ✅ Improved audio for easier listening ✅ Silences removed to save you time Q&A at start, Elon's speech starts at 1h 25m. FYI, in this edit I put the Q&A first but at the event, the Q&A followed the speech. Separate edits are coming with just the speech (and) one with just the Q&A. Share them with someone you know!

stevenmarkryan

1,017,484 次观看 • 1 年前

How to Create Music That Matches Your Video's Sound with Suno Studio I made a quick video showing how I use Suno Studio to generate background music that fits the audio and mood of a video, similar to the example in the quoted post. This is the method I use at the moment but if you have a better approach, feel free to share it. Before starting, export the original audio from your video. I usually do this in CapCut. After generating a track you like in Suno Studio, you can import it back into CapCut and add it to your video. *This is not a paid partnership.

How to Create Music That Matches Your Video's Sound with Suno Studio I made a quick video showing how I use Suno Studio to generate background music that fits the audio and mood of a video, similar to the example in the quoted post. This is the method I use at the moment but if you have a better approach, feel free to share it. Before starting, export the original audio from your video. I usually do this in CapCut. After generating a track you like in Suno Studio, you can import it back into CapCut and add it to your video. *This is not a paid partnership.

Kōda

37,597 次观看 • 2 个月前

Seed Audio 1.0 is now available on Runway for all paid plans. Generate up to 120 seconds of dynamic speech, sound design and music with simple text prompts. Try it now at the link below.

Seed Audio 1.0 is now available on Runway for all paid plans. Generate up to 120 seconds of dynamic speech, sound design and music with simple text prompts. Try it now at the link below.

Runway

32,302 次观看 • 2 个月前

If you want some definitive proof that ABC destroyed the audio in Biden's interview, here you are. Left: original audio Right: denoised audio Bidens mic was EQ'd to pick up as much low end and high end ambient noise as possible, which is why his voice sounds so much more distorted than Stephanopoulos. Even with noise removal, the audio was beyond salvaging. You can listen for yourself. Biden team, if you're listening, never do another network interview/speech unless your own audio engineers get to set up the equipment.

If you want some definitive proof that ABC destroyed the audio in Biden's interview, here you are. Left: original audio Right: denoised audio Bidens mic was EQ'd to pick up as much low end and high end ambient noise as possible, which is why his voice sounds so much more distorted than Stephanopoulos. Even with noise removal, the audio was beyond salvaging. You can listen for yourself. Biden team, if you're listening, never do another network interview/speech unless your own audio engineers get to set up the equipment.

I Smoked The Clippers

4,765,075 次观看 • 2 年前

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏

Lior Alexander

231,665 次观看 • 3 年前

Tried more 🎙️ AI voice cloning of my own voice to see if I can make into an audio book with I used a 5 min audio sample from a podcast with only my own voice cut It sounds a bit too American/Canadian for me but 90% close I think

Tried more 🎙️ AI voice cloning of my own voice to see if I can make into an audio book with I used a 5 min audio sample from a podcast with only my own voice cut It sounds a bit too American/Canadian for me but 90% close I think

@levelsio

144,119 次观看 • 2 年前

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

Google presents AudioPaLM: A Large Language Model That Can Speak and Listen paper page: introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2 [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified multimodal architecture that can process and generate text and speech with applications including speech recognition and speech-to-speech translation. AudioPaLM inherits the capability to preserve paralinguistic information such as speaker identity and intonation from AudioLM and the linguistic knowledge present only in text large language models such as PaLM-2. We demonstrate that initializing AudioPaLM with the weights of a text-only large language model improves speech processing, successfully leveraging the larger quantity of text training data used in pretraining to assist with the speech tasks. The resulting model significantly outperforms existing systems for speech translation tasks and has the ability to perform zero-shot speech-to-text translation for many languages for which input/target language combinations were not seen in training. AudioPaLM also demonstrates features of audio language models, such as transferring a voice across languages based on a short spoken prompt.

AK

290,517 次观看 • 3 年前

🇫🇷🇺🇸 BRIGITTE MACRON: "FREE SPEECH NEEDS LIMITS" "I do know that I am the United States, a country of free speech. But I think that liberty does have some rules. We have to be free but adapt to certain codes."

🇫🇷🇺🇸 BRIGITTE MACRON: "FREE SPEECH NEEDS LIMITS" "I do know that I am the United States, a country of free speech. But I think that liberty does have some rules. We have to be free but adapt to certain codes."

NewsForce

964,077 次观看 • 5 个月前

Droid made me a little ghost. I can hotkey anywhere, it pops up, it hears my audio, turns it into text, passes it to any model to do tool calls, or manage my screen, then responds in audio. Unfortunately the audio of the TTS isn't in the video, but it's using chatterbox

Droid made me a little ghost. I can hotkey anywhere, it pops up, it hears my audio, turns it into text, passes it to any model to do tool calls, or manage my screen, then responds in audio. Unfortunately the audio of the TTS isn't in the video, but it's using chatterbox

0xSero

10,122 次观看 • 4 个月前

Q: I enjoy listening to music with my headphones 🎧 How can I avoid hearing loss? A: 📶 Keep the volume below 60% of max on your device 🎧 Use well-fitted, noise-cancelling headphones. This can reduce the need to raise volume 🔊 Reduce time spent listening to loud sounds

Q: I enjoy listening to music with my headphones 🎧 How can I avoid hearing loss? A: 📶 Keep the volume below 60% of max on your device 🎧 Use well-fitted, noise-cancelling headphones. This can reduce the need to raise volume 🔊 Reduce time spent listening to loud sounds

World Health Organization (WHO)

53,724 次观看 • 1 年前

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

Today we released Meta Spirit LM — our first open source multimodal language model that freely mixes text and speech. Many existing AI voice experiences today use ASR to techniques to process speech before synthesizing with an LLM to generate text — but these approaches compromise the expressive aspects of speech. Using phonetic, pitch and tone tokens, Spirit LM models can overcome these limitations for both inputs and outputs to generate more natural sounding speech while also learning new tasks across ASR, TTS and speech classification. We hope that sharing this work will enable the research community to further new approaches for text and speech integration.

AI at Meta

351,764 次观看 • 1 年前