Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

🥳Excited to share our latest work: "Diff-A-Riff"! 🥁 A Latent Diffusion Model that generates instrumental accompaniments for any musical input, specifically tailored for music producers! It's faster, lighter, and produces superior audio quality. Control via text/audio references. 48kHz sample rate, (pseudo) stereo, ~3Gb memory, takes 6 seconds to generate... 90 seconds of music. Trained on a single GPU. 📜 🎶 🎸 "Diff-A-Riff" adapts to any musical input, following the artist's unique style. 🎛️ Optional controls via text prompts, audio references, interpolation slider, pseudo-stereo width and loop intensity. 🎚️ It produces state-of-the-art audio quality indistinguishable from real data by human raters and operates at unprecedented speed. 🧠 "Diff-A-Riff" is smaller and more efficient than previous models thanks to its Consistency Autoencoder, making it accessible and practical for various applications. Big shoutout to my outer space colleagues: Javier Nistal, the Machine in "machine learning" 🚄, Marco Pasini, the neural net whisperer 🤫, Cyran Aouameur, the troubleshootah 🛠️, Maarten Grachten, aka MaartenGPT 🤖. #Teamwork #AI #MusicTech #Innovation Javier Nistal Marco Pasini Cyran Aouameur SonyCSL(Paris)_Music Teamshow more

Stefan Lattner

1,635 subscribers

17,760 Aufrufe • vor 2 Jahren •via X (Twitter)

Wissenschaft & Technologie Musik

Anya Rossi• Live Now

Private livecam show

10 Kommentare

Profilbild von Stefan Lattner

Stefan Lattnervor 2 Jahren

Update! 🥳 This example demonstrates how to produce a song with Diff-A-Riff starting from a simple template and adding one instrument at a time:

Profilbild von ARGO

ARGOvor 2 Jahren

Can't wait to kick this around. So glad to see all the new accomapaniment stuff coming out. This conrner of the tech. Accompaniment, trained musician agents, working towards realtime interactive "jamming" to vibe with automation that doesn't feel like just playing with a looper!

Profilbild von Michael Anslow

Michael Anslowvor 2 Jahren

Very cool stuff!

Profilbild von Devin Arne

Devin Arnevor 2 Jahren

Congratulations! This is an amazing tool, the one stem at a time approach is great for musicians/producers as it gives granular control. I could also see this as helpful for film/TV composers to make variations on themes and cues. Hope to be able to use this in the near future!

Profilbild von Janne Spijkervet

Janne Spijkervetvor 2 Jahren

Now this is exciting!! Well done! Two questions came to mind after a first glance: 1) 50% dropout on audio/CLAP conditioning appeared high to me, was this required? And 2) the impact of the CAE (I’m impressed by the decoder at that number of params!) Nice work!

Profilbild von Florian van Oirschot

Florian van Oirschotvor 2 Jahren

Wow! That sounds impressive :)

Profilbild von Matan Gover

Matan Govervor 2 Jahren

Awesome work, great results!! One question about inpainting - in the demo audios, the inpainting result sounds slightly different than the original, even in the non-masked region. Is this due to CAE roundtrip?

Profilbild von SGM

SGMvor 2 Jahren

@deeplearnmusic Interesting I would love and appreciate to see a similar type of work with tts please look into it Thankyou so much

Profilbild von gregorylent

gregorylentvor 2 Jahren

@MacTuitui can i mix the output? delete tracks? change levels ??

Profilbild von Stefan Lattner

Stefan Lattnervor 2 Jahren

@MacTuitui Hi Gregory! The model adds one track at a time, giving the user full control over the mix.

Ähnliche Videos

🥳 We present our #ICASSP2024 paper: A diffusion model that generates production-quality (bass) audio stems to any audio input. 😎🎸 According to our experience, that's more useful to artists than generating full mixes. 🙃 📜Paper: 🎶Demo: by Marco Pasini 👈💪SonyCSL(Paris)_Music Team #MusicAI

🥳 We present our #ICASSP2024 paper: A diffusion model that generates production-quality (bass) audio stems to any audio input. 😎🎸 According to our experience, that's more useful to artists than generating full mixes. 🙃 📜Paper: 🎶Demo: by Marco Pasini 👈💪SonyCSL(Paris)_Music Team #MusicAI

Stefan Lattner

20,287 Aufrufe • vor 2 Jahren

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏

Lior Alexander

231,729 Aufrufe • vor 3 Jahren

🎥 Today we’re premiering Meta Movie Gen: the most advanced media foundation models to-date. Developed by AI research teams at Meta, Movie Gen delivers state-of-the-art results across a range of capabilities. We’re excited for the potential of this line of research to usher in entirely new possibilities for casual creators and creative professionals alike. More details and examples of what Movie Gen can do ➡️ 🛠️ Movie Gen models and capabilities Movie Gen Video: 30B parameter transformer model that can generate high-quality and high-definition images and videos from a single text prompt. Movie Gen Audio: A 13B parameter transformer model that can take a video input along with optional text prompts for controllability to generate high-fidelity audio synced to the video. It can generate ambient sound, instrumental background music and foley sound — delivering state-of-the-art results in audio quality, video-to-audio alignment and text-to-audio alignment. Precise video editing: Using a generated or existing video and accompanying text instructions as an input it can perform localized edits such as adding, removing or replacing elements — or global changes like background or style changes. Personalized videos: Using an image of a person and a text prompt, the model can generate a video with state-of-the-art results on character preservation and natural movement in video. We’re continuing to work closely with creative professionals from across the field to integrate their feedback as we work towards a potential release. We look forward to sharing more on this work and the creative possibilities it will enable in the future.

🎥 Today we’re premiering Meta Movie Gen: the most advanced media foundation models to-date. Developed by AI research teams at Meta, Movie Gen delivers state-of-the-art results across a range of capabilities. We’re excited for the potential of this line of research to usher in entirely new possibilities for casual creators and creative professionals alike. More details and examples of what Movie Gen can do ➡️ 🛠️ Movie Gen models and capabilities Movie Gen Video: 30B parameter transformer model that can generate high-quality and high-definition images and videos from a single text prompt. Movie Gen Audio: A 13B parameter transformer model that can take a video input along with optional text prompts for controllability to generate high-fidelity audio synced to the video. It can generate ambient sound, instrumental background music and foley sound — delivering state-of-the-art results in audio quality, video-to-audio alignment and text-to-audio alignment. Precise video editing: Using a generated or existing video and accompanying text instructions as an input it can perform localized edits such as adding, removing or replacing elements — or global changes like background or style changes. Personalized videos: Using an image of a person and a text prompt, the model can generate a video with state-of-the-art results on character preservation and natural movement in video. We’re continuing to work closely with creative professionals from across the field to integrate their feedback as we work towards a potential release. We look forward to sharing more on this work and the creative possibilities it will enable in the future.

AI at Meta

2,267,756 Aufrufe • vor 2 Jahren

As part of Meta Movie Gen, we trained a 13B parameter audio generation model that can take a video + optional text prompts to generate high quality audio — including ambient sound, foley & instrumental background music — all synced to the video. Details ➡️

As part of Meta Movie Gen, we trained a 13B parameter audio generation model that can take a video + optional text prompts to generate high quality audio — including ambient sound, foley & instrumental background music — all synced to the video. Details ➡️

AI at Meta

76,729 Aufrufe • vor 2 Jahren

MusicHiFi Fast High-Fidelity Stereo Vocoding Diffusion-based audio and music generation models commonly generate music by constructing an image representation of audio (e.g., a mel-spectrogram) and then converting it to audio using a phase reconstruction model or vocoder.

MusicHiFi Fast High-Fidelity Stereo Vocoding Diffusion-based audio and music generation models commonly generate music by constructing an image representation of audio (e.g., a mel-spectrogram) and then converting it to audio using a phase reconstruction model or vocoder.

AK

27,285 Aufrufe • vor 2 Jahren

Researchers at Meta recently shared MAGNeT, a single non-autoregressive transformer model for text-to-music & text-to-sound generation capable of generating audio on-par with the quality of SOTA models — at 7x the speed. MAGNeT is open source as part of AudioCraft. Hear audio samples and get more details on this work ➡️

Researchers at Meta recently shared MAGNeT, a single non-autoregressive transformer model for text-to-music & text-to-sound generation capable of generating audio on-par with the quality of SOTA models — at 7x the speed. MAGNeT is open source as part of AudioCraft. Hear audio samples and get more details on this work ➡️

AI at Meta

122,001 Aufrufe • vor 2 Jahren

My Artist Brain is now available for download. 🧠 Combobulator by DataMind Audio is a real-time neural audio synthesis plugin. Input any audio signal, and the AI, ethically sourced and trained on artist-created datasets, will recreate the timbre in real-time.

My Artist Brain is now available for download. 🧠 Combobulator by DataMind Audio is a real-time neural audio synthesis plugin. Input any audio signal, and the AI, ethically sourced and trained on artist-created datasets, will recreate the timbre in real-time.

Mr. Bill

38,403 Aufrufe • vor 2 Jahren

Seed-Music A Unified Framework for High Quality and Controlled Music Generation discuss: We introduce Seed-Music, a suite of music generation systems capable of producing high-quality music with fine-grained style control. Our unified framework leverages both auto-regressive language modeling and diffusion approaches to support two key music creation workflows: controlled music generation and post-production editing. For controlled music generation, our system enables vocal music generation with performance controls from multi-modal inputs, including style descriptions, audio references, musical scores, and voice prompts. For post-production editing, it offers interactive tools for editing lyrics and vocal melodies directly in the generated audio.

Seed-Music A Unified Framework for High Quality and Controlled Music Generation discuss: We introduce Seed-Music, a suite of music generation systems capable of producing high-quality music with fine-grained style control. Our unified framework leverages both auto-regressive language modeling and diffusion approaches to support two key music creation workflows: controlled music generation and post-production editing. For controlled music generation, our system enables vocal music generation with performance controls from multi-modal inputs, including style descriptions, audio references, musical scores, and voice prompts. For post-production editing, it offers interactive tools for editing lyrics and vocal melodies directly in the generated audio.

AK

32,967 Aufrufe • vor 2 Jahren

Today we’re launching Stable Audio 2.5: The first audio model built for enterprise-grade sound production 🔊 Audio influences brand engagement by 86%, but few enterprises are leveraging audio as an extension of their brand, making customized sound an untapped differentiator. Stable Audio 2.5 is purpose-built for this opportunity to create customizable, high-quality audio at scale, with capabilities that include: ▶️ Improved musical composition: Generate full songs with multi-part structure, meaning a clear intro, middle, and outro. ▶️ Audio inpainting: Input audio, select where the track should start, and the model uses the context to generate the rest of the track. ▶️ Customization: Our team can fine-tune Stable Audio 2.5 to help enterprises create the right sound for their brand. ▶️ Faster inference: The model can generate up to three-minute long tracks in under two seconds on a GPU, outputting in just eight steps (compared to ~50 in the previous model). You can learn more here 👉

Stability AI

68,341 Aufrufe • vor 1 Jahr

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

AK

143,557 Aufrufe • vor 3 Jahren

Type a sentence, get any sound - from talking cats to singing saxophones. Brilliant release by NVIDIA ✨ NVIDIA just unveiled Fugatto, a groundbreaking 2.5B parameter audio AI model that can generate and transform any combination of music, voices, and sounds using text prompts and audio inputs Fugatto could ultimately allow developers and creators to bring sounds to life simply by inputting text prompts, → The model demonstrates unique capabilities like creating hybrid sounds (trumpet barking), changing accents/emotions in voices, and allowing fine-grained control over sound transitions - trained on millions of audio samples using 32 NVIDIA H100 GPUs 👨‍🔧 Architecture Built as a foundational generative transformer model leveraging NVIDIA's previous work in speech modeling and audio understanding. The training process involved creating a specialized blended dataset containing millions of audio samples → ComposableART's Innovation in Audio Control Introduces a novel technique allowing combination of instructions that were only seen separately during training. Users can blend different audio attributes and control their intensity → Temporal Interpolation Capabilities Enables generation of evolving soundscapes with precise control over transitions. Can create dynamic audio sequences like rainstorms fading into birdsong at dawn → Processes both text and audio inputs flexibly, enabling tasks like removing instruments from songs or modifying specific audio characteristics while preserving others → Shows capabilities beyond its training data, creating entirely new sound combinations through interaction between different trained abilities 🔍 Real-world Applications → Allows rapid prototyping of musical ideas, style experimentation, and real-time sound creation during studio sessions → Enables dynamic audio asset generation matching gameplay situations, reducing pre-recorded audio requirements → Can modify voice characteristics for language learning applications, allowing content delivery in familiar voices NVIDIA AI Developer

Type a sentence, get any sound - from talking cats to singing saxophones. Brilliant release by NVIDIA ✨ NVIDIA just unveiled Fugatto, a groundbreaking 2.5B parameter audio AI model that can generate and transform any combination of music, voices, and sounds using text prompts and audio inputs Fugatto could ultimately allow developers and creators to bring sounds to life simply by inputting text prompts, → The model demonstrates unique capabilities like creating hybrid sounds (trumpet barking), changing accents/emotions in voices, and allowing fine-grained control over sound transitions - trained on millions of audio samples using 32 NVIDIA H100 GPUs 👨‍🔧 Architecture Built as a foundational generative transformer model leveraging NVIDIA's previous work in speech modeling and audio understanding. The training process involved creating a specialized blended dataset containing millions of audio samples → ComposableART's Innovation in Audio Control Introduces a novel technique allowing combination of instructions that were only seen separately during training. Users can blend different audio attributes and control their intensity → Temporal Interpolation Capabilities Enables generation of evolving soundscapes with precise control over transitions. Can create dynamic audio sequences like rainstorms fading into birdsong at dawn → Processes both text and audio inputs flexibly, enabling tasks like removing instruments from songs or modifying specific audio characteristics while preserving others → Shows capabilities beyond its training data, creating entirely new sound combinations through interaction between different trained abilities 🔍 Real-world Applications → Allows rapid prototyping of musical ideas, style experimentation, and real-time sound creation during studio sessions → Enables dynamic audio asset generation matching gameplay situations, reducing pre-recorded audio requirements → Can modify voice characteristics for language learning applications, allowing content delivery in familiar voices NVIDIA AI Developer

Rohan Paul

96,354 Aufrufe • vor 1 Jahr

Stereo depth models usually force a tradeoff. 📌 Paper + Code Either strong zero-shot generalization or real-time speed. Fast-FoundationStereo closes that gap. The model accelerates FoundationStereo by more than 10× while keeping comparable depth quality, making real-time stereo matching practical for robotics and XR. On an RTX 3090, it produces dense metric depth maps and 3D point clouds in real time, outperforming existing real-time stereo approaches and even challenging some heavier zero-shot models. The key idea is a divide-and-conquer architecture that removes bottlenecks in common stereo matching components while preserving the strengths of the original teacher model. Training also tackles a major constraint in stereo vision: the lack of ground-truth metric depth data. The team built an automatic data curation pipeline that generates pseudo-labels from internet-scale stereo imagery using the Stereo4D dataset. Result: a real-time stereo foundation model with strong zero-shot performance and practical runtime speed. Code, pretrained weights, and pseudo-labeled stereo data will be released. Thanks for sharing, Bowen Wen! 📍Paper: Code: Project: ——- Weekly robotics and AI insights. Subscribe free:

Stereo depth models usually force a tradeoff. 📌 Paper + Code Either strong zero-shot generalization or real-time speed. Fast-FoundationStereo closes that gap. The model accelerates FoundationStereo by more than 10× while keeping comparable depth quality, making real-time stereo matching practical for robotics and XR. On an RTX 3090, it produces dense metric depth maps and 3D point clouds in real time, outperforming existing real-time stereo approaches and even challenging some heavier zero-shot models. The key idea is a divide-and-conquer architecture that removes bottlenecks in common stereo matching components while preserving the strengths of the original teacher model. Training also tackles a major constraint in stereo vision: the lack of ground-truth metric depth data. The team built an automatic data curation pipeline that generates pseudo-labels from internet-scale stereo imagery using the Stereo4D dataset. Result: a real-time stereo foundation model with strong zero-shot performance and practical runtime speed. Code, pretrained weights, and pseudo-labeled stereo data will be released. Thanks for sharing, Bowen Wen! 📍Paper: Code: Project: ——- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

11,914 Aufrufe • vor 2 Monaten

LFM2-Audio just dropped! It's a 1.5B model that understands and generates both text and audio Inference 10x faster + quality on par with models 10x larger Available today on Hugging Face and our playground 🥳

LFM2-Audio just dropped! It's a 1.5B model that understands and generates both text and audio Inference 10x faster + quality on par with models 10x larger Available today on Hugging Face and our playground 🥳

Maxime Labonne

85,053 Aufrufe • vor 1 Jahr

Wan2.5: Let Sound Take the Director’s Chair! 🎬 Today, we’re excited to unveil another major feature in our powerful Wan 2.5 Preview: Native Audio-Driven Video Generation. ✨ Now you can use audio input directly for both text-to-video and image-to-video generation. Combine audio with text prompts or a reference image to shape your video's narrative. ✨ With support for videos up to 10 seconds and enhanced video quality, unlock a richer visual space where more engaging stories come to life.

Wan2.5: Let Sound Take the Director’s Chair! 🎬 Today, we’re excited to unveil another major feature in our powerful Wan 2.5 Preview: Native Audio-Driven Video Generation. ✨ Now you can use audio input directly for both text-to-video and image-to-video generation. Combine audio with text prompts or a reference image to shape your video's narrative. ✨ With support for videos up to 10 seconds and enhanced video quality, unlock a richer visual space where more engaging stories come to life.

Wan

52,172 Aufrufe • vor 1 Jahr

HOLY FUCK! Zyphra just dropped Zonos - Apache 2.0 licensed, Multilingual, Text to Speech model with INSTANT voice cloning! 🔥 > Zero-shot TTS with Voice Cloning: Input text and a 10-30 second speaker sample to generate high-quality text-to-speech output > Audio Prefix Inputs: Enhance speaker matching by adding an audio prefix to the text, enabling behaviors like whispering that are hard to achieve with voice cloning alone > Multilingual Support: Supports English, Japanese, Chinese, French, and German > Audio Quality & Emotion Control: Fine-tune speaking rate, pitch, frequency, audio quality, and emotions (e.g., happiness, anger, sadness, fear) > Fast Performance: Runs at ~2x real-time speed on an RTX 4090 > Available on the Hugging Face Hub 🤗

HOLY FUCK! Zyphra just dropped Zonos - Apache 2.0 licensed, Multilingual, Text to Speech model with INSTANT voice cloning! 🔥 > Zero-shot TTS with Voice Cloning: Input text and a 10-30 second speaker sample to generate high-quality text-to-speech output > Audio Prefix Inputs: Enhance speaker matching by adding an audio prefix to the text, enabling behaviors like whispering that are hard to achieve with voice cloning alone > Multilingual Support: Supports English, Japanese, Chinese, French, and German > Audio Quality & Emotion Control: Fine-tune speaking rate, pitch, frequency, audio quality, and emotions (e.g., happiness, anger, sadness, fear) > Fast Performance: Runs at ~2x real-time speed on an RTX 4090 > Available on the Hugging Face Hub 🤗

Vaibhav (VB) Srivastav

299,066 Aufrufe • vor 1 Jahr

We’re excited to share our groundbreaking new audio-driven performance model, featuring hyper-real expressions in near real-time. Any length video, in any style, is ready in 6 seconds or less—in HD. And we’ve managed to make it 20x faster and cheaper 💅 It’s all part of our vision to make AI video accessible and fun for more people—a way for them to connect, not just create. Rolling out soon in the Pika social app.

We’re excited to share our groundbreaking new audio-driven performance model, featuring hyper-real expressions in near real-time. Any length video, in any style, is ready in 6 seconds or less—in HD. And we’ve managed to make it 20x faster and cheaper 💅 It’s all part of our vision to make AI video accessible and fun for more people—a way for them to connect, not just create. Rolling out soon in the Pika social app.

Pika

1,244,154 Aufrufe • vor 1 Jahr

Introducing Stable Audio 2.0 – a new model capable of producing high-quality, full tracks with coherent musical structure up to three minutes long at 44.1 kHz stereo from a single prompt. Explore the model and start creating for free at: Read the blogpost here: (1/3)

Introducing Stable Audio 2.0 – a new model capable of producing high-quality, full tracks with coherent musical structure up to three minutes long at 44.1 kHz stereo from a single prompt. Explore the model and start creating for free at: Read the blogpost here: (1/3)

Stability AI

458,450 Aufrufe • vor 2 Jahren

3DTopia-XL High-Quality 3D PBR Asset Generation via Primitive Diffusion demo: model: 3DTopia-XL scales high-quality 3D asset generation using Diffusion Transformer (DiT) built upon an expressive and efficient 3D representation, PrimX. The denoising process takes 5 seconds to generate a 3D PBR asset from text/image input which is ready for the graphics pipeline to use.

3DTopia-XL High-Quality 3D PBR Asset Generation via Primitive Diffusion demo: model: 3DTopia-XL scales high-quality 3D asset generation using Diffusion Transformer (DiT) built upon an expressive and efficient 3D representation, PrimX. The denoising process takes 5 seconds to generate a 3D PBR asset from text/image input which is ready for the graphics pipeline to use.

AK

87,282 Aufrufe • vor 2 Jahren