Загрузка видео...

Не удалось загрузить видео

На главную

🥳Excited to share our latest work: "Diff-A-Riff"! 🥁 A Latent Diffusion Model that generates instrumental accompaniments for any musical input, specifically tailored for music producers! It's faster, lighter, and produces superior audio quality. Control via text/audio references. 48kHz sample rate, (pseudo) stereo, ~3Gb memory, takes 6 seconds to generate...

deeplearnmusic's profile picture

Stefan Lattner

1,603 subscribers

17,703 просмотров • 2 лет назад •via X (Twitter)

Наука и технологииМузыка
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 10

Фото профиля Stefan Lattner
Stefan Lattner2 лет назад

Update! 🥳 This example demonstrates how to produce a song with Diff-A-Riff starting from a simple template and adding one instrument at a time:

Фото профиля ARGO
ARGO2 лет назад

Can't wait to kick this around. So glad to see all the new accomapaniment stuff coming out. This conrner of the tech. Accompaniment, trained musician agents, working towards realtime interactive "jamming" to vibe with automation that doesn't feel like just playing with a looper!

Фото профиля Michael Anslow
Michael Anslow2 лет назад

Very cool stuff!

Фото профиля Devin Arne
Devin Arne2 лет назад

Congratulations! This is an amazing tool, the one stem at a time approach is great for musicians/producers as it gives granular control. I could also see this as helpful for film/TV composers to make variations on themes and cues. Hope to be able to use this in the near future!

Фото профиля Janne Spijkervet
Janne Spijkervet2 лет назад

Now this is exciting!! Well done! Two questions came to mind after a first glance: 1) 50% dropout on audio/CLAP conditioning appeared high to me, was this required? And 2) the impact of the CAE (I’m impressed by the decoder at that number of params!) Nice work!

Фото профиля Florian van Oirschot
Florian van Oirschot2 лет назад

Wow! That sounds impressive :)

Фото профиля Matan Gover
Matan Gover2 лет назад

Awesome work, great results!! One question about inpainting - in the demo audios, the inpainting result sounds slightly different than the original, even in the non-masked region. Is this due to CAE roundtrip?

Фото профиля SGM
SGM2 лет назад

@deeplearnmusic Interesting I would love and appreciate to see a similar type of work with tts please look into it Thankyou so much

Фото профиля gregorylent
gregorylent2 лет назад

@MacTuitui can i mix the output? delete tracks? change levels ??

Фото профиля Stefan Lattner
Stefan Lattner2 лет назад

@MacTuitui Hi Gregory! The model adds one track at a time, giving the user full control over the mix.

Похожие видео

🎥 Today we’re premiering Meta Movie Gen: the most advanced media foundation models to-date. Developed by AI research teams at Meta, Movie Gen delivers state-of-the-art results across a range of capabilities. We’re excited for the potential of this line of research to usher in entirely new possibilities for casual creators and creative professionals alike. More details and examples of what Movie Gen can do ➡️ 🛠️ Movie Gen models and capabilities Movie Gen Video: 30B parameter transformer model that can generate high-quality and high-definition images and videos from a single text prompt. Movie Gen Audio: A 13B parameter transformer model that can take a video input along with optional text prompts for controllability to generate high-fidelity audio synced to the video. It can generate ambient sound, instrumental background music and foley sound — delivering state-of-the-art results in audio quality, video-to-audio alignment and text-to-audio alignment. Precise video editing: Using a generated or existing video and accompanying text instructions as an input it can perform localized edits such as adding, removing or replacing elements — or global changes like background or style changes. Personalized videos: Using an image of a person and a text prompt, the model can generate a video with state-of-the-art results on character preservation and natural movement in video. We’re continuing to work closely with creative professionals from across the field to integrate their feedback as we work towards a potential release. We look forward to sharing more on this work and the creative possibilities it will enable in the future.
0:30

🎥 Today we’re premiering Meta Movie Gen: the most advanced media foundation models to-date. Developed by AI research teams at Meta, Movie Gen delivers state-of-the-art results across a range of capabilities. We’re excited for the potential of this line of research to usher in entirely new possibilities for casual creators and creative professionals alike. More details and examples of what Movie Gen can do ➡️ 🛠️ Movie Gen models and capabilities Movie Gen Video: 30B parameter transformer model that can generate high-quality and high-definition images and videos from a single text prompt. Movie Gen Audio: A 13B parameter transformer model that can take a video input along with optional text prompts for controllability to generate high-fidelity audio synced to the video. It can generate ambient sound, instrumental background music and foley sound — delivering state-of-the-art results in audio quality, video-to-audio alignment and text-to-audio alignment. Precise video editing: Using a generated or existing video and accompanying text instructions as an input it can perform localized edits such as adding, removing or replacing elements — or global changes like background or style changes. Personalized videos: Using an image of a person and a text prompt, the model can generate a video with state-of-the-art results on character preservation and natural movement in video. We’re continuing to work closely with creative professionals from across the field to integrate their feedback as we work towards a potential release. We look forward to sharing more on this work and the creative possibilities it will enable in the future.

AI at Meta

2,264,719 просмотров • 1 год назад

Type a sentence, get any sound - from talking cats to singing saxophones. Brilliant release by NVIDIA ✨ NVIDIA just unveiled Fugatto, a groundbreaking 2.5B parameter audio AI model that can generate and transform any combination of music, voices, and sounds using text prompts and audio inputs Fugatto could ultimately allow developers and creators to bring sounds to life simply by inputting text prompts, → The model demonstrates unique capabilities like creating hybrid sounds (trumpet barking), changing accents/emotions in voices, and allowing fine-grained control over sound transitions - trained on millions of audio samples using 32 NVIDIA H100 GPUs 👨‍🔧 Architecture Built as a foundational generative transformer model leveraging NVIDIA's previous work in speech modeling and audio understanding. The training process involved creating a specialized blended dataset containing millions of audio samples → ComposableART's Innovation in Audio Control Introduces a novel technique allowing combination of instructions that were only seen separately during training. Users can blend different audio attributes and control their intensity → Temporal Interpolation Capabilities Enables generation of evolving soundscapes with precise control over transitions. Can create dynamic audio sequences like rainstorms fading into birdsong at dawn → Processes both text and audio inputs flexibly, enabling tasks like removing instruments from songs or modifying specific audio characteristics while preserving others → Shows capabilities beyond its training data, creating entirely new sound combinations through interaction between different trained abilities 🔍 Real-world Applications → Allows rapid prototyping of musical ideas, style experimentation, and real-time sound creation during studio sessions → Enables dynamic audio asset generation matching gameplay situations, reducing pre-recorded audio requirements → Can modify voice characteristics for language learning applications, allowing content delivery in familiar voices NVIDIA AI Developer
3:03

Type a sentence, get any sound - from talking cats to singing saxophones. Brilliant release by NVIDIA ✨ NVIDIA just unveiled Fugatto, a groundbreaking 2.5B parameter audio AI model that can generate and transform any combination of music, voices, and sounds using text prompts and audio inputs Fugatto could ultimately allow developers and creators to bring sounds to life simply by inputting text prompts, → The model demonstrates unique capabilities like creating hybrid sounds (trumpet barking), changing accents/emotions in voices, and allowing fine-grained control over sound transitions - trained on millions of audio samples using 32 NVIDIA H100 GPUs 👨‍🔧 Architecture Built as a foundational generative transformer model leveraging NVIDIA's previous work in speech modeling and audio understanding. The training process involved creating a specialized blended dataset containing millions of audio samples → ComposableART's Innovation in Audio Control Introduces a novel technique allowing combination of instructions that were only seen separately during training. Users can blend different audio attributes and control their intensity → Temporal Interpolation Capabilities Enables generation of evolving soundscapes with precise control over transitions. Can create dynamic audio sequences like rainstorms fading into birdsong at dawn → Processes both text and audio inputs flexibly, enabling tasks like removing instruments from songs or modifying specific audio characteristics while preserving others → Shows capabilities beyond its training data, creating entirely new sound combinations through interaction between different trained abilities 🔍 Real-world Applications → Allows rapid prototyping of musical ideas, style experimentation, and real-time sound creation during studio sessions → Enables dynamic audio asset generation matching gameplay situations, reducing pre-recorded audio requirements → Can modify voice characteristics for language learning applications, allowing content delivery in familiar voices NVIDIA AI Developer

Rohan Paul

96,354 просмотров • 1 год назад

Today is a good day for open science. As part of our continued commitment to the growth and development of an open ecosystem, today at Meta FAIR we’re announcing four new publicly available AI models and additional research artifacts to inspire innovation in the community and help advance AI in a responsible way. More in the video from Joelle Pineau. What we’re releasing: 🦎 Meta Chameleon 7B & 34B language models that support mixed-modal input and text-only outputs. 🪙 Meta Multi-Token Prediction Pretrained Language Models for code completion using Multi-Token Prediction. 🎼 Meta JASCO Generative text-to-music models capable of accepting various conditioning inputs for greater controllability. Paper available today with a pretrained model coming soon. 🗣️ Meta AudioSeal An audio watermarking model that we believe is the first designed specifically for the localized detection of AI-generated speech, available under a commercial license. 📝 Additional RAI artifacts Including research, data and code to measure and improve the representation of geographical and cultural preferences and diversity in AI systems. We believe that access to state-of-the-art AI creates opportunities for everyone – not just a small handful of Big Tech companies. We’re excited to share this work and to see how the community learns, iterates and builds using this technology. Details and access to everything released by FAIR today ➡️
1:14

Today is a good day for open science. As part of our continued commitment to the growth and development of an open ecosystem, today at Meta FAIR we’re announcing four new publicly available AI models and additional research artifacts to inspire innovation in the community and help advance AI in a responsible way. More in the video from Joelle Pineau. What we’re releasing: 🦎 Meta Chameleon 7B & 34B language models that support mixed-modal input and text-only outputs. 🪙 Meta Multi-Token Prediction Pretrained Language Models for code completion using Multi-Token Prediction. 🎼 Meta JASCO Generative text-to-music models capable of accepting various conditioning inputs for greater controllability. Paper available today with a pretrained model coming soon. 🗣️ Meta AudioSeal An audio watermarking model that we believe is the first designed specifically for the localized detection of AI-generated speech, available under a commercial license. 📝 Additional RAI artifacts Including research, data and code to measure and improve the representation of geographical and cultural preferences and diversity in AI systems. We believe that access to state-of-the-art AI creates opportunities for everyone – not just a small handful of Big Tech companies. We’re excited to share this work and to see how the community learns, iterates and builds using this technology. Details and access to everything released by FAIR today ➡️

AI at Meta

380,751 просмотров • 2 лет назад

This is probably the most complex workflow I’ve ever built, only with open-source tools. It took my 4 days. It takes four inputs: author, title, and style; and generates a full visual animated story in one click in ComfyUI . I worked on it for four days. There are still some bugs, but here’s the first preview. Here’s a quick breakdown: - The four inputs are sent to LLMs with precise instructions to generate: first, prompts for images and image modifications; second, prompts for animations; third, prompts for generating music. - All voices are generated from the text and timed precisely, as they determine the length of each animation segment. - The first image and video are generated to serve as the title, but also as the guide for all other images created for the video. - Titles and subtitles are also added automatically in Comfy. - I also developed a lot of custom nodes for minor frame calculations, mostly to match audio and video. - The full system is a large loop that, for each line of text, generates an image and then a video from that image. The loop was the hardest part to build in this workflow, so it can process either a 20-second video or a 2-minute video with the same input. - There are multiple combinations of LLMs that try to understand the text in the best way to provide the best prompts for images and video. - The final video is assembled entirely within ComfyUI. - The music is generated based on the LLM output and matches the exact timing of the full animation. - Done! For reference, this workflow uses a lot of models and only works on an RTX 6000 Pro with plenty of RAM. My goal is not to replace humans, as I’ll try to explain later, this workflow is highly controlled and can be adapted or reworked at any point by real artists! My aim was to create a tool that can animate text in one go, allowing the AI some freedom while keeping a strict flow. I don’t know yet how I’ll share this workflow with people, I still need to polish it properly, but maybe through Patreon. Anyway, I hope you enjoy my research, and let’s always keep pushing further! :)
0:55

This is probably the most complex workflow I’ve ever built, only with open-source tools. It took my 4 days. It takes four inputs: author, title, and style; and generates a full visual animated story in one click in ComfyUI . I worked on it for four days. There are still some bugs, but here’s the first preview. Here’s a quick breakdown: - The four inputs are sent to LLMs with precise instructions to generate: first, prompts for images and image modifications; second, prompts for animations; third, prompts for generating music. - All voices are generated from the text and timed precisely, as they determine the length of each animation segment. - The first image and video are generated to serve as the title, but also as the guide for all other images created for the video. - Titles and subtitles are also added automatically in Comfy. - I also developed a lot of custom nodes for minor frame calculations, mostly to match audio and video. - The full system is a large loop that, for each line of text, generates an image and then a video from that image. The loop was the hardest part to build in this workflow, so it can process either a 20-second video or a 2-minute video with the same input. - There are multiple combinations of LLMs that try to understand the text in the best way to provide the best prompts for images and video. - The final video is assembled entirely within ComfyUI. - The music is generated based on the LLM output and matches the exact timing of the full animation. - Done! For reference, this workflow uses a lot of models and only works on an RTX 6000 Pro with plenty of RAM. My goal is not to replace humans, as I’ll try to explain later, this workflow is highly controlled and can be adapted or reworked at any point by real artists! My aim was to create a tool that can animate text in one go, allowing the AI some freedom while keeping a strict flow. I don’t know yet how I’ll share this workflow with people, I still need to polish it properly, but maybe through Patreon. Anyway, I hope you enjoy my research, and let’s always keep pushing further! :)

Lovis Odin

58,571 просмотров • 9 месяцев назад

Meet My AI Ears. A lot of folks ask me how I capture ASMR video and audio for training of AI? I always use Binaural 3D audio and have for decades in different forms. But how? A Brief History of Binaural Recording Binaural recording, the foundation of 3D audio, dates back to 1881 when French inventor Clément Ader created the first system using multiple telephone transmitters at the Paris Opera to transmit stereo sound to listeners, simulating spatial presence. By the 1920s, patents like W. Bartlett Jones’ 1927 filing advanced devices for capturing and reproducing “binaural” signals. The 1930s saw Alan Blumlein’s work on stereophonic sound, which he termed “binaural,” laying groundwork for modern stereo. Commercial milestones hit in the 1950s with binaural records from labels like Cook Laboratories and the first binaural reel-to-reel tapes. A resurgence came in the 1970s with Neumann’s KU-80 dummy head, the first commercial binaural system. Today, it’s integral to VR, ASMR, and immersive media. The Technology of Binaural 3D Audio At its core, binaural recording mimics human hearing by using two microphones placed in ear-shaped molds or a dummy head, separated like human ears (typically 14-18 cm apart). This captures spatial cues: interaural time differences (ITD) for sound arrival timing, interaural level differences (ILD) for volume variations, and head-related transfer functions (HRTF) that account for how the head, torso, and pinnae filter sounds. The result? A 3D soundscape that tricks the brain into perceiving direction, distance, and elevation when played back via headphones—no speakers needed for immersion. Advanced setups use omnidirectional capsules (e.g., DPA 4060) for high-fidelity capture, often in silicone ears to replicate natural diffraction. I use the 3DIO Microphones today but I would cover a dummy head in texture material and place two stereo (4 channels) microphones in each ear. I would then mix down the resulting signals into stereo. The 3DIO series features dual omnidirectional capsules in realistic silicone ear molds, spaced 14 cm apart for compact, accurate 3D capture based on over 13 years of research into human hearing. Models like the Free Space Pro II use premium DPA 4060 CORE capsules for ultra-low noise and high sensitivity, delivering stereo output ideal for immersive applications like game audio. Today just about any ASMR producer uses these. But I use them to capture, curate and archive sound and video we will lose or just about lost for AI training in a way no model or AI company is doing today. I am duplicating the human 3D binaural audio experience and memory. Below is a crude demonstration. If you can listen in headphones. Or turn your phone sideways to feel the audio space. I’ll have a far more professional demo soon to show the real power of 3D audio. (Oh that music is a MIDI player that uses disks to play).
0:35

Meet My AI Ears. A lot of folks ask me how I capture ASMR video and audio for training of AI? I always use Binaural 3D audio and have for decades in different forms. But how? A Brief History of Binaural Recording Binaural recording, the foundation of 3D audio, dates back to 1881 when French inventor Clément Ader created the first system using multiple telephone transmitters at the Paris Opera to transmit stereo sound to listeners, simulating spatial presence. By the 1920s, patents like W. Bartlett Jones’ 1927 filing advanced devices for capturing and reproducing “binaural” signals. The 1930s saw Alan Blumlein’s work on stereophonic sound, which he termed “binaural,” laying groundwork for modern stereo. Commercial milestones hit in the 1950s with binaural records from labels like Cook Laboratories and the first binaural reel-to-reel tapes. A resurgence came in the 1970s with Neumann’s KU-80 dummy head, the first commercial binaural system. Today, it’s integral to VR, ASMR, and immersive media. The Technology of Binaural 3D Audio At its core, binaural recording mimics human hearing by using two microphones placed in ear-shaped molds or a dummy head, separated like human ears (typically 14-18 cm apart). This captures spatial cues: interaural time differences (ITD) for sound arrival timing, interaural level differences (ILD) for volume variations, and head-related transfer functions (HRTF) that account for how the head, torso, and pinnae filter sounds. The result? A 3D soundscape that tricks the brain into perceiving direction, distance, and elevation when played back via headphones—no speakers needed for immersion. Advanced setups use omnidirectional capsules (e.g., DPA 4060) for high-fidelity capture, often in silicone ears to replicate natural diffraction. I use the 3DIO Microphones today but I would cover a dummy head in texture material and place two stereo (4 channels) microphones in each ear. I would then mix down the resulting signals into stereo. The 3DIO series features dual omnidirectional capsules in realistic silicone ear molds, spaced 14 cm apart for compact, accurate 3D capture based on over 13 years of research into human hearing. Models like the Free Space Pro II use premium DPA 4060 CORE capsules for ultra-low noise and high sensitivity, delivering stereo output ideal for immersive applications like game audio. Today just about any ASMR producer uses these. But I use them to capture, curate and archive sound and video we will lose or just about lost for AI training in a way no model or AI company is doing today. I am duplicating the human 3D binaural audio experience and memory. Below is a crude demonstration. If you can listen in headphones. Or turn your phone sideways to feel the audio space. I’ll have a far more professional demo soon to show the real power of 3D audio. (Oh that music is a MIDI player that uses disks to play).

Brian Roemmele

30,800 просмотров • 6 месяцев назад

XAI EXTENDS GROK IMAGINE VIDEO GENERATION TO 10 SECONDS WITH QUALITY ENHANCEMENTS xAI has updated its Grok Imagine tool to produce videos lasting 10 seconds, doubling the prior limit. This change, along with refinements in visual and audio elements, expands the tool's utility for short-form content creation. xAI released the upgrade to Grok Imagine in early 2026. The company, founded by Elon Musk, announced the feature through posts on the X platform. This follows previous iterations where videos were capped at shorter durations, typically 5 seconds. Grok Imagine allows users to generate videos from text prompts, building on its image creation capabilities. The update addresses constraints in video length that limited expressive potential. Users can now input descriptions to create clips, such as animations or scenes, without needing initial images. This positions the tool within the broader landscape of AI-driven multimodal generation, where text-to-video systems are increasingly common. The core adjustment doubles the maximum video duration from 5 seconds to 10 seconds. Accompanying this are upgrades to video quality, including more stable visuals, richer details, and improved clarity. Audio has also been enhanced for better output, making the generated content more immersive. These changes were described as "big improvements across the board" in the announcement. No specific benchmarks or quantitative metrics for the quality improvements were detailed in the release statements. The feature rollout appears gradual, with some users accessing it via the Grok app or web interface. xAI has not introduced user controls for exact timing, though such options are mentioned as future possibilities. This development highlights xAI's emphasis on iterative enhancements in generative AI tools. By extending duration while refining output fidelity, it reflects engineering priorities aimed at balancing computational efficiency with user needs. The focus on audio and visual stability suggests attention to common pitfalls in early text-to-video models, such as inconsistencies or artifacts. The sources do not specify the underlying model architecture changes or training data adjustments enabling this upgrade. Performance in real-world scenarios, like handling complex prompts or maintaining consistency across clips, remains unquantified in the announcements. Interpretations of broader implications for AI video generation would require additional evidence beyond what's provided.
0:29

XAI EXTENDS GROK IMAGINE VIDEO GENERATION TO 10 SECONDS WITH QUALITY ENHANCEMENTS xAI has updated its Grok Imagine tool to produce videos lasting 10 seconds, doubling the prior limit. This change, along with refinements in visual and audio elements, expands the tool's utility for short-form content creation. xAI released the upgrade to Grok Imagine in early 2026. The company, founded by Elon Musk, announced the feature through posts on the X platform. This follows previous iterations where videos were capped at shorter durations, typically 5 seconds. Grok Imagine allows users to generate videos from text prompts, building on its image creation capabilities. The update addresses constraints in video length that limited expressive potential. Users can now input descriptions to create clips, such as animations or scenes, without needing initial images. This positions the tool within the broader landscape of AI-driven multimodal generation, where text-to-video systems are increasingly common. The core adjustment doubles the maximum video duration from 5 seconds to 10 seconds. Accompanying this are upgrades to video quality, including more stable visuals, richer details, and improved clarity. Audio has also been enhanced for better output, making the generated content more immersive. These changes were described as "big improvements across the board" in the announcement. No specific benchmarks or quantitative metrics for the quality improvements were detailed in the release statements. The feature rollout appears gradual, with some users accessing it via the Grok app or web interface. xAI has not introduced user controls for exact timing, though such options are mentioned as future possibilities. This development highlights xAI's emphasis on iterative enhancements in generative AI tools. By extending duration while refining output fidelity, it reflects engineering priorities aimed at balancing computational efficiency with user needs. The focus on audio and visual stability suggests attention to common pitfalls in early text-to-video models, such as inconsistencies or artifacts. The sources do not specify the underlying model architecture changes or training data adjustments enabling this upgrade. Performance in real-world scenarios, like handling complex prompts or maintaining consistency across clips, remains unquantified in the announcements. Interpretations of broader implications for AI video generation would require additional evidence beyond what's provided.

Lacey

28,881 просмотров • 5 месяцев назад

Suno's mission is to make it possible for everyone to make music. We imagine a future where music is a bigger, more valuable, and more meaningful part of people's lives than it even is today. Technology enables a future where the whole world can explore, create, and be active participants in an art form most have only ever consumed. From professional musicians seeking inspiration to friends and family writing songs for each other, we are exploring new ways to create, listen to, and experience music. So far, more than 12 million people are engaging with music in new ways that wouldn't be possible without Suno. We see this as early but promising progress. Major record labels see this vision as a threat to their business. Each and every time there's been innovation in music — from the earliest forms of recorded music, to sampling, to drum machines, to remixing, MP3s, and streaming music — the record labels have attempted to limit progress. They have spent decades attempting to control the terms of how we create and enjoy music, and this time is no different. So, it is perhaps not a surprise that on June 24th, members of the Recording Industry Association of America, which represents the major record labels, filed a lawsuit against Suno, alleging that the data used in training our music generation technologies infringed on the copyrights of the major record labels that they represent. This lawsuit is fundamentally flawed on both the facts and the law, and is nothing more than yet another instance where they chose litigation over innovation. For starters, the major record labels clearly hold misconceptions about how our technology works. Suno helps people create music through a similar process to one humans have used forever: by learning styles, patterns, and forms (in essence, the "grammar" or music), and then inventing new music around them. The major record labels are trying to argue that neural networks are mere parrots — copying and repeating — when in reality model training looks a lot more like a kid learning to write new rock songs by listening religiously to rock music. Like that kid, Suno gets better the more our AI learns. We train our models on medium- and high-quality music we can find on the open internet — just as Google's Gemini, Microsoft's Copilot, Anthropic's Claude, OpenAI's ChatGPT, and even Apple's new Apple Intelligence train their models on the open internet. Much of the open internet indeed contains copyrighted materials, and some of it is owned by major record labels. But, just like the kid writing their own rock songs after listening to the genre — or a teacher or a journalist reviewing existing materials to draw new insights — learning is not infringing. It never has been, and it is not now. The timing of this lawsuit was somewhat surprising. When this lawsuit landed, Suno was, in fact, having productive discussions with a number of the RIAA's major record label members to find ways of expanding the pie for music together. We did so not because we had to, but because we believe that the music industry could help us lead this expansion of opportunity for everyone, rather than resisting it. Whether this lawsuit is the result of over-eager lawyers throwing their weight around, or a conscious strategy to gain leverage in our commercial discussions, we believe that this lawsuit is an unnecessary impediment to a larger and more valuable future for music. This is particularly the case because Suno is a new kind of musical instrument, one that enables a new kind of creative process for everyone and opens new business opportunities for the industry. Suno is designed for original music, and we prize originality, both in how we build our product and in how people use it. People who use Suno are using the product to create their own, original music. They are not trying to recreate an existing song that can be heard somewhere else on the internet for free. But, even if they were trying to copy existing music, we have myriad controls in place to encourage originality and prevent duplicative use cases. We do so more aggressively than any other company in the industry, including other startups. Some of our originality-guarding features include checking for and preventing copyrighted content in audio uploads, and disallowing artist-based descriptions in requests to generate music. Why do we work to encourage originality? We do this because it makes for a more fun and engaging experience to create entirely original compositions on Suno. We do it because we think it makes Suno incredibly valuable to be a place where new musical talent can shine. AI allows anyone to realize the songs in their head, regardless of the money, equipment, or connections that they have. The future is an explosion of new artists that are creating music in new ways, building fan bases, finding new reasons to smile, and getting famous. We hope that the major record labels realize that we can build a stronger foundation for the music industry of tomorrow, together. With or without them, we will continue pursuing our mission on behalf of the many millions of music fans already creating with Suno, and all those who will in the coming months. We are excited and humbled to support this next generation of musicians and the music they create. --- 🎥: shing86 jams with Suno
0:28

Suno's mission is to make it possible for everyone to make music. We imagine a future where music is a bigger, more valuable, and more meaningful part of people's lives than it even is today. Technology enables a future where the whole world can explore, create, and be active participants in an art form most have only ever consumed. From professional musicians seeking inspiration to friends and family writing songs for each other, we are exploring new ways to create, listen to, and experience music. So far, more than 12 million people are engaging with music in new ways that wouldn't be possible without Suno. We see this as early but promising progress. Major record labels see this vision as a threat to their business. Each and every time there's been innovation in music — from the earliest forms of recorded music, to sampling, to drum machines, to remixing, MP3s, and streaming music — the record labels have attempted to limit progress. They have spent decades attempting to control the terms of how we create and enjoy music, and this time is no different. So, it is perhaps not a surprise that on June 24th, members of the Recording Industry Association of America, which represents the major record labels, filed a lawsuit against Suno, alleging that the data used in training our music generation technologies infringed on the copyrights of the major record labels that they represent. This lawsuit is fundamentally flawed on both the facts and the law, and is nothing more than yet another instance where they chose litigation over innovation. For starters, the major record labels clearly hold misconceptions about how our technology works. Suno helps people create music through a similar process to one humans have used forever: by learning styles, patterns, and forms (in essence, the "grammar" or music), and then inventing new music around them. The major record labels are trying to argue that neural networks are mere parrots — copying and repeating — when in reality model training looks a lot more like a kid learning to write new rock songs by listening religiously to rock music. Like that kid, Suno gets better the more our AI learns. We train our models on medium- and high-quality music we can find on the open internet — just as Google's Gemini, Microsoft's Copilot, Anthropic's Claude, OpenAI's ChatGPT, and even Apple's new Apple Intelligence train their models on the open internet. Much of the open internet indeed contains copyrighted materials, and some of it is owned by major record labels. But, just like the kid writing their own rock songs after listening to the genre — or a teacher or a journalist reviewing existing materials to draw new insights — learning is not infringing. It never has been, and it is not now. The timing of this lawsuit was somewhat surprising. When this lawsuit landed, Suno was, in fact, having productive discussions with a number of the RIAA's major record label members to find ways of expanding the pie for music together. We did so not because we had to, but because we believe that the music industry could help us lead this expansion of opportunity for everyone, rather than resisting it. Whether this lawsuit is the result of over-eager lawyers throwing their weight around, or a conscious strategy to gain leverage in our commercial discussions, we believe that this lawsuit is an unnecessary impediment to a larger and more valuable future for music. This is particularly the case because Suno is a new kind of musical instrument, one that enables a new kind of creative process for everyone and opens new business opportunities for the industry. Suno is designed for original music, and we prize originality, both in how we build our product and in how people use it. People who use Suno are using the product to create their own, original music. They are not trying to recreate an existing song that can be heard somewhere else on the internet for free. But, even if they were trying to copy existing music, we have myriad controls in place to encourage originality and prevent duplicative use cases. We do so more aggressively than any other company in the industry, including other startups. Some of our originality-guarding features include checking for and preventing copyrighted content in audio uploads, and disallowing artist-based descriptions in requests to generate music. Why do we work to encourage originality? We do this because it makes for a more fun and engaging experience to create entirely original compositions on Suno. We do it because we think it makes Suno incredibly valuable to be a place where new musical talent can shine. AI allows anyone to realize the songs in their head, regardless of the money, equipment, or connections that they have. The future is an explosion of new artists that are creating music in new ways, building fan bases, finding new reasons to smile, and getting famous. We hope that the major record labels realize that we can build a stronger foundation for the music industry of tomorrow, together. With or without them, we will continue pursuing our mission on behalf of the many millions of music fans already creating with Suno, and all those who will in the coming months. We are excited and humbled to support this next generation of musicians and the music they create. --- 🎥: shing86 jams with Suno

Suno

46,808 просмотров • 1 год назад

For generative AI to become an interesting art tool, we need much more control over the output. The slot-machine-like nature of pure text-to-image leaves too much to chance. Using the "Real-time Latent Consistency Model" that I'm using in the example here, is the first time I truly got a glimpse of a future where we'll be able to use our artistic skills and sensibility, to get control over AI image gen. Systems like these will never be able to match the quality or originality of a skilled artist, it won't surprise us in the same way an artist can. Things are a mess in terms of the training data these models are based on, and the questions about copyright concerns and about a time when everything will look the same are very valid. At some point capabilities like these will be embedded in photoshop, and anyone will be able to generate a pretty picture. But to create interesting designs, to tell original stories and to surprise us, we need creatives and artists with something on their mind. We'll be able to create immersive worlds, by making brush-strokes and sculpt marks, without needing to worry about all the dials, plugins, wires of our 3d and 2d tools today. I love to sculpt, I love to draw, and I love to explore new mediums and new ways to create. The Gen AI tools we have today are far from perfect, and things need to be steered in a better direction. For that we need artists to help point the way. Gen AI isn't going away.. it's too powerful and has the potential to allow us to tell stories like never before. Like all other big technological shifts, tech like this will come at a cost, but it will also open up new opportunities and empower a new generation of storytellers. I might be naive, but I believe that human ingenuity and creativity will persevere in this new world ♥️ #art #ai
0:32

For generative AI to become an interesting art tool, we need much more control over the output. The slot-machine-like nature of pure text-to-image leaves too much to chance. Using the "Real-time Latent Consistency Model" that I'm using in the example here, is the first time I truly got a glimpse of a future where we'll be able to use our artistic skills and sensibility, to get control over AI image gen. Systems like these will never be able to match the quality or originality of a skilled artist, it won't surprise us in the same way an artist can. Things are a mess in terms of the training data these models are based on, and the questions about copyright concerns and about a time when everything will look the same are very valid. At some point capabilities like these will be embedded in photoshop, and anyone will be able to generate a pretty picture. But to create interesting designs, to tell original stories and to surprise us, we need creatives and artists with something on their mind. We'll be able to create immersive worlds, by making brush-strokes and sculpt marks, without needing to worry about all the dials, plugins, wires of our 3d and 2d tools today. I love to sculpt, I love to draw, and I love to explore new mediums and new ways to create. The Gen AI tools we have today are far from perfect, and things need to be steered in a better direction. For that we need artists to help point the way. Gen AI isn't going away.. it's too powerful and has the potential to allow us to tell stories like never before. Like all other big technological shifts, tech like this will come at a cost, but it will also open up new opportunities and empower a new generation of storytellers. I might be naive, but I believe that human ingenuity and creativity will persevere in this new world ♥️ #art #ai

Martin Nebelong

1,660,017 просмотров • 2 лет назад

As many of you know, for the last five months I've been working full-time on my next big thing. The challenge was to invent something new and implement it entirely using LLMs for writing code. The first stage of the project is now complete: the web application, which I called is now online and accepting users. You can see a short demo in the video. 100% of the code of the app was generated by LLMs (mostly Gemini and Claude, maybe 10% of ChatGPT). I haven't written a single line of code. The tech stack is TypeScript, React, and Supabase/Postgres which was (and still is) fully new to me. During these five months, I implemented from scratch three versions of the software. It started as a Markdown editor to help me with my book writing and ended up as an AI-assisted reading and self-learning platform. What makes ChapterPal unique is a novel reading experience where the user can use the keyboard keys to reveal or "unreveal" the content and ask questions at any moment. (Mouse wheel, touchpad, smartphone screen, and voice input are also supported.) The LLM receives the entire content of the chapter and tries to answer questions based on the chapter's content, which reduces the chance of hallucination to the minimum. (Though not to 0%, of course, but near it.) This way of content consumption is known as **active reading,** a strategy for engaging with a text to improve comprehension and retention by consciously interacting with the material. The goal is to move beyond passive reading to a deeper understanding of the text and to remember key information more effectively. The registration on ChapterPal is via the waiting list. This is to avoid unexpected load spikes and cloud charges. Usually, it takes less than 24 hours for me to activate a user. Give it a try and let me know what you think. The next stage is finishing the content ingestion pipeline, which will automatically convert high-quality content from sources like HTML, PDF, and LaTeX into Markdown. Obviously, only those pieces whose licenses allow creating copies. ChapterPal has its own collection of textbooks and articles on AI, machine learning, and data science topics. If you don't find a piece of content you would like to read in ChapterPal's collection, a Chrome extension, ChapterPal Uploader, allows you to upload any PDF or HTML page to ChapterPal in one click. The content is only available for you to read to avoid the possibility of copyright infringement. I hope you enjoy using it as much as I enjoy building it.
3:41

As many of you know, for the last five months I've been working full-time on my next big thing. The challenge was to invent something new and implement it entirely using LLMs for writing code. The first stage of the project is now complete: the web application, which I called is now online and accepting users. You can see a short demo in the video. 100% of the code of the app was generated by LLMs (mostly Gemini and Claude, maybe 10% of ChatGPT). I haven't written a single line of code. The tech stack is TypeScript, React, and Supabase/Postgres which was (and still is) fully new to me. During these five months, I implemented from scratch three versions of the software. It started as a Markdown editor to help me with my book writing and ended up as an AI-assisted reading and self-learning platform. What makes ChapterPal unique is a novel reading experience where the user can use the keyboard keys to reveal or "unreveal" the content and ask questions at any moment. (Mouse wheel, touchpad, smartphone screen, and voice input are also supported.) The LLM receives the entire content of the chapter and tries to answer questions based on the chapter's content, which reduces the chance of hallucination to the minimum. (Though not to 0%, of course, but near it.) This way of content consumption is known as **active reading,** a strategy for engaging with a text to improve comprehension and retention by consciously interacting with the material. The goal is to move beyond passive reading to a deeper understanding of the text and to remember key information more effectively. The registration on ChapterPal is via the waiting list. This is to avoid unexpected load spikes and cloud charges. Usually, it takes less than 24 hours for me to activate a user. Give it a try and let me know what you think. The next stage is finishing the content ingestion pipeline, which will automatically convert high-quality content from sources like HTML, PDF, and LaTeX into Markdown. Obviously, only those pieces whose licenses allow creating copies. ChapterPal has its own collection of textbooks and articles on AI, machine learning, and data science topics. If you don't find a piece of content you would like to read in ChapterPal's collection, a Chrome extension, ChapterPal Uploader, allows you to upload any PDF or HTML page to ChapterPal in one click. The content is only available for you to read to avoid the possibility of copyright infringement. I hope you enjoy using it as much as I enjoy building it.

BURKOV

81,180 просмотров • 8 месяцев назад

This is one-shot assembly: you show examples of what to build, and the robot just does it. (see original post: To share more on how this works, the robot is controlled in real time by a neural network that takes in video pixels and outputs 100Hz actions. The video below is part of the raw input passed directly into the model. I also like this view (at 1x speed) because it shows more of the (I think very cool) subtle moments of dexterity near the fingertips 👌 One-shot assembly seemed like a dream even just a year ago — it's not easy. It requires both the high-level reasoning of "what to build" (recognizing the geometry of the structures presented by the human), and the low-level visuomotor control of "how to build it" (purposefully re-orienting individual pieces and nudging them together in place). While possible to manually engineer a complex system for this (e.g. w/ hierarchical control, or explicit state representations), we were curious if our own Foundation model could do it all end-to-end with just some post-training data. Surprisingly, it just worked. Nothing about the recipe is substantially different than any other demo we’ve run in the past, and we’re excited about its implications on model capabilities: • On contextual reasoning, these models can (i) attend to task-related pixels in the peripheral view of the video inputs, and (ii) retain this knowledge in-context while ignoring irrelevant background. This is useful for generalizing to a wide range of real workflows: e.g. paying attention to what’s coming down the conveyor line, or glancing at the instructions displayed on a nearby monitor. • On dexterity, these models can produce contact-rich "commonsense" behaviors that can be difficult to pre-program or write language instructions for e.g. rolling a brick slightly to align its studs against the bottom of another, re-grasping to get a better grip or to move out of the way before a forceful press, or gently pushing the corners of a brick against the mat to rotate it in hand and stand it up vertically (i.e. extrinsic dexterity). These aspects work together to form a capability that resembles fast adaptation — a hallmark of intelligence, relevant for real use cases. This has also expanded my own perspective on what's possible with robot learning, using a recipe that's repeatable for many more skills. This milestone stands on top of the solid technical foundations we’ve built here at Generalist: hardcore controls & hardware, all in-house built models, and a data engine that "just works." We're a small group of hyper-focused engineers, and hands-down the highest talent-density team I’ve ever worked with. We're accelerating and scaling aggressively towards unlocking next-generation robot intelligence. Building Legos is just one example, and it's clear to me that we're headed towards a future where robots can do just about anything we want them to. Its coming, and we're going to make it happen.
1:01

This is one-shot assembly: you show examples of what to build, and the robot just does it. (see original post: To share more on how this works, the robot is controlled in real time by a neural network that takes in video pixels and outputs 100Hz actions. The video below is part of the raw input passed directly into the model. I also like this view (at 1x speed) because it shows more of the (I think very cool) subtle moments of dexterity near the fingertips 👌 One-shot assembly seemed like a dream even just a year ago — it's not easy. It requires both the high-level reasoning of "what to build" (recognizing the geometry of the structures presented by the human), and the low-level visuomotor control of "how to build it" (purposefully re-orienting individual pieces and nudging them together in place). While possible to manually engineer a complex system for this (e.g. w/ hierarchical control, or explicit state representations), we were curious if our own Foundation model could do it all end-to-end with just some post-training data. Surprisingly, it just worked. Nothing about the recipe is substantially different than any other demo we’ve run in the past, and we’re excited about its implications on model capabilities: • On contextual reasoning, these models can (i) attend to task-related pixels in the peripheral view of the video inputs, and (ii) retain this knowledge in-context while ignoring irrelevant background. This is useful for generalizing to a wide range of real workflows: e.g. paying attention to what’s coming down the conveyor line, or glancing at the instructions displayed on a nearby monitor. • On dexterity, these models can produce contact-rich "commonsense" behaviors that can be difficult to pre-program or write language instructions for e.g. rolling a brick slightly to align its studs against the bottom of another, re-grasping to get a better grip or to move out of the way before a forceful press, or gently pushing the corners of a brick against the mat to rotate it in hand and stand it up vertically (i.e. extrinsic dexterity). These aspects work together to form a capability that resembles fast adaptation — a hallmark of intelligence, relevant for real use cases. This has also expanded my own perspective on what's possible with robot learning, using a recipe that's repeatable for many more skills. This milestone stands on top of the solid technical foundations we’ve built here at Generalist: hardcore controls & hardware, all in-house built models, and a data engine that "just works." We're a small group of hyper-focused engineers, and hands-down the highest talent-density team I’ve ever worked with. We're accelerating and scaling aggressively towards unlocking next-generation robot intelligence. Building Legos is just one example, and it's clear to me that we're headed towards a future where robots can do just about anything we want them to. Its coming, and we're going to make it happen.

Andy Zeng

49,317 просмотров • 9 месяцев назад

This will retire 90% of RAG systems with dignity (and a sad song playlist). Powered by DSPy: If you're still building "text in, text out" chatbots that only perform blind vector and text searches, you're not gonna make it! My team just dropped Elysia, and it's not just an incremental successor to Verba… It's a whole rethink of how we interact with our data using AI. 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗘𝗹𝘆𝗶𝘀𝗮? An open-source platform for building agentic RAG architectures. It learns from your preferences, intelligently categorizes, labels, and searches through your data, and provides complete transparency into its decision-making process. The long & exciting feature list: • 𝗧𝗿𝗮𝗻𝘀𝗽𝗮𝗿𝗲𝗻𝘁 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻-𝗧𝗿𝗲𝗲 𝗔𝗴𝗲𝗻𝘁𝘀: Elysia’s core is a customizable decision tree, and it visualizes its entire reasoning process, showing you why it chooses a specific tool or path. It enables advanced error handling, self-healing from failed queries, and prevents infinite loops. You can also add custom tools and branches to build complex, state-aware workflows. • 𝗗𝗮𝘁𝗮 𝗔𝘄𝗮𝗿𝗲𝗻𝗲𝘀𝘀: Before it even attempts a query, Elysia performs a full analysis of your data collections. This eliminates the blind search problem plaguing most RAG systems and allows for far more complex and accurate query generation. • 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗗𝗮𝘁𝗮 𝗗𝗶𝘀𝗽𝗹𝗮𝘆𝘀: Your RAG pipeline shouldn't be limited to text, right? That’s why Elysia analyzes each query's results and chooses the best way to display them, from tables and charts to product cards and GitHub tickets. It also features a comprehensive data explorer with search, sorting, and filtering capabilities. • 𝗛𝘆𝗽𝗲𝗿-𝗣𝗲𝗿𝘀𝗼𝗻𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝘃𝗶𝗮 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸: It uses your positively-rated queries as few-shot examples to improve future responses. This allows you to use smaller, faster models that perform like larger ones over time, cutting costs without sacrificing quality for most use cases. • 𝗖𝗵𝘂𝗻𝗸-𝗢𝗻-𝗗𝗲𝗺𝗮𝗻𝗱: Elysia chunks documents at query time. It performs initial searches on document-level vectors and only chunks relevant documents on the fly, storing them in a parallel quantized collection with cross references for future use. 𝗧𝗵𝗲 𝗦𝘁𝗮𝗰𝗸 Elysia is built from scratch on Weaviate, using its native features like named vectors, a variety of search types, filters, cross references, quantization, etc. It uses DSPy for LLM interactions and is delivered as a production-ready application via FastAPI, serving a NextJS frontend as static HTML. Also available as a Python package via pip: 𝗽𝗶𝗽 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 𝗲𝗹𝘆𝘀𝗶𝗮-𝗮𝗶 Type: 𝗲𝗹𝘆𝘀𝗶𝗮 𝘀𝘁𝗮𝗿𝘁 Connect your Weaviate cluster and go explore what’s possible.
0:35

This will retire 90% of RAG systems with dignity (and a sad song playlist). Powered by DSPy: If you're still building "text in, text out" chatbots that only perform blind vector and text searches, you're not gonna make it! My team just dropped Elysia, and it's not just an incremental successor to Verba… It's a whole rethink of how we interact with our data using AI. 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗘𝗹𝘆𝗶𝘀𝗮? An open-source platform for building agentic RAG architectures. It learns from your preferences, intelligently categorizes, labels, and searches through your data, and provides complete transparency into its decision-making process. The long & exciting feature list: • 𝗧𝗿𝗮𝗻𝘀𝗽𝗮𝗿𝗲𝗻𝘁 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻-𝗧𝗿𝗲𝗲 𝗔𝗴𝗲𝗻𝘁𝘀: Elysia’s core is a customizable decision tree, and it visualizes its entire reasoning process, showing you why it chooses a specific tool or path. It enables advanced error handling, self-healing from failed queries, and prevents infinite loops. You can also add custom tools and branches to build complex, state-aware workflows. • 𝗗𝗮𝘁𝗮 𝗔𝘄𝗮𝗿𝗲𝗻𝗲𝘀𝘀: Before it even attempts a query, Elysia performs a full analysis of your data collections. This eliminates the blind search problem plaguing most RAG systems and allows for far more complex and accurate query generation. • 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗗𝗮𝘁𝗮 𝗗𝗶𝘀𝗽𝗹𝗮𝘆𝘀: Your RAG pipeline shouldn't be limited to text, right? That’s why Elysia analyzes each query's results and chooses the best way to display them, from tables and charts to product cards and GitHub tickets. It also features a comprehensive data explorer with search, sorting, and filtering capabilities. • 𝗛𝘆𝗽𝗲𝗿-𝗣𝗲𝗿𝘀𝗼𝗻𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝘃𝗶𝗮 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸: It uses your positively-rated queries as few-shot examples to improve future responses. This allows you to use smaller, faster models that perform like larger ones over time, cutting costs without sacrificing quality for most use cases. • 𝗖𝗵𝘂𝗻𝗸-𝗢𝗻-𝗗𝗲𝗺𝗮𝗻𝗱: Elysia chunks documents at query time. It performs initial searches on document-level vectors and only chunks relevant documents on the fly, storing them in a parallel quantized collection with cross references for future use. 𝗧𝗵𝗲 𝗦𝘁𝗮𝗰𝗸 Elysia is built from scratch on Weaviate, using its native features like named vectors, a variety of search types, filters, cross references, quantization, etc. It uses DSPy for LLM interactions and is delivered as a production-ready application via FastAPI, serving a NextJS frontend as static HTML. Also available as a Python package via pip: 𝗽𝗶𝗽 𝗶𝗻𝘀𝘁𝗮𝗹𝗹 𝗲𝗹𝘆𝘀𝗶𝗮-𝗮𝗶 Type: 𝗲𝗹𝘆𝘀𝗶𝗮 𝘀𝘁𝗮𝗿𝘁 Connect your Weaviate cluster and go explore what’s possible.

Philip Vollet

93,598 просмотров • 11 месяцев назад

Sora 2 has the capabilities to make an entire anime episode with time and effort. This is a turning point for generative AI models and it is terrifying to think what is next to come. The difference between Sora and Sora 2 is absolutely staggering. Yes, this looks cool. I put the time into it to make it as close to art as possible. This was not simple to make and not just prompting and throwing it in some editor. Every frame is generated of another scene and even the reference image for characters I used to start this project were AI generated. This was all created with 10 second clips pulled apart and chunked out into the final product. Regardless of the imagery you see, I did not directly use any artist's art for this video. This 10-minute video was generated over the course of a week from more than 700 text prompts. It was built on technology trained by scraping the uncredited, uncompensated contributions of countless human artists and animators. The creation of this single video consumed an enormous amount of energy and water, equivalent to powering a home for days and requiring hundreds of liters of fresh water for cooling. But that is nothing. In the 10 minutes you spend watching this, the global network of AI video generators will create over 6,250 more short videos. The combined energy required for that 10 minutes of global creation is enough to power an average household for nearly 2 years. - It's a double-edged sword. While AI uses immense amount of energy has clear immoral issues with scalping the hard work of artists, this also provides those a medium who have potentially spent their life attempting to draw out the ideas in their head and failing to grasp it. My sister is an artist, I grew up always attempting to draw but never could get the image in my head on paper, I've spent the better part of over 20 years to teach myself through watching videos and practice, however I don't have steady hands and frankly have just been unable to make any vision come to life. Which brings me to the conflict here. This would have been a dream of mine to be able to get frames created and flesh out the story in my head. To make this video professionally, it would be an absurd amount of money. A small studio by itself would be over a million dollars just to start up. To hire a studio would be most likely over ~200k. I would love to make this story I have in my head through legit and traditional means, I would love to start a Kickstarter to get funding and hire artists, voice actors, and a production team. However I know that would most likely be an impossible reach that would further fuel the hate. Regardless, I understand the worry this causes. The fear this produces. However, we can't just ignore where AI is currently at and just tell people not to use it or even worse threaten people who use it. People will always use the shiny new toy in front of them, so the real question is how do we either make it work for us and work along side it, or how do we ACTUALLY implement a method to protect art rather than tell people to not use it. Respect your artists. Review their ToS and don't upload their hard work to a model without permission.
10:33

Sora 2 has the capabilities to make an entire anime episode with time and effort. This is a turning point for generative AI models and it is terrifying to think what is next to come. The difference between Sora and Sora 2 is absolutely staggering. Yes, this looks cool. I put the time into it to make it as close to art as possible. This was not simple to make and not just prompting and throwing it in some editor. Every frame is generated of another scene and even the reference image for characters I used to start this project were AI generated. This was all created with 10 second clips pulled apart and chunked out into the final product. Regardless of the imagery you see, I did not directly use any artist's art for this video. This 10-minute video was generated over the course of a week from more than 700 text prompts. It was built on technology trained by scraping the uncredited, uncompensated contributions of countless human artists and animators. The creation of this single video consumed an enormous amount of energy and water, equivalent to powering a home for days and requiring hundreds of liters of fresh water for cooling. But that is nothing. In the 10 minutes you spend watching this, the global network of AI video generators will create over 6,250 more short videos. The combined energy required for that 10 minutes of global creation is enough to power an average household for nearly 2 years. - It's a double-edged sword. While AI uses immense amount of energy has clear immoral issues with scalping the hard work of artists, this also provides those a medium who have potentially spent their life attempting to draw out the ideas in their head and failing to grasp it. My sister is an artist, I grew up always attempting to draw but never could get the image in my head on paper, I've spent the better part of over 20 years to teach myself through watching videos and practice, however I don't have steady hands and frankly have just been unable to make any vision come to life. Which brings me to the conflict here. This would have been a dream of mine to be able to get frames created and flesh out the story in my head. To make this video professionally, it would be an absurd amount of money. A small studio by itself would be over a million dollars just to start up. To hire a studio would be most likely over ~200k. I would love to make this story I have in my head through legit and traditional means, I would love to start a Kickstarter to get funding and hire artists, voice actors, and a production team. However I know that would most likely be an impossible reach that would further fuel the hate. Regardless, I understand the worry this causes. The fear this produces. However, we can't just ignore where AI is currently at and just tell people not to use it or even worse threaten people who use it. People will always use the shiny new toy in front of them, so the real question is how do we either make it work for us and work along side it, or how do we ACTUALLY implement a method to protect art rather than tell people to not use it. Respect your artists. Review their ToS and don't upload their hard work to a model without permission.

LUͦʷCͦk

18,417 просмотров • 9 месяцев назад

NVIDIA just unleashed SANA-WM and it’s an absolute MONSTER for the future of open source AI! A blazing-fast 2.6B-parameter open-source world model that doesn’t just generate video… it creates controllable, physics-rich, high-fidelity worlds on demand. Why this is insanely powerful: • One image + text prompt + 6-DoF camera trajectory → generates 720p videos up to 60 seconds long with buttery-smooth, precisely controlled camera movement. You’re not just watching, you’re piloting the simulation. • Runs locally on a single consumer GPU (RTX 5090 level) thanks to heavy distillation + NVFP4 quantization. Full 60-second clip denoised in ~34 seconds. No massive clusters required. • 36× higher throughput than previous open models while rivaling (or beating) closed industrial giants in visual quality and consistency. • Trained lightning-fast: ~213K public videos in just 15 days on 64 H100s. • Built with next-level tech: Hybrid Linear Attention, dual-branch camera control, two-stage pipeline, and rock-solid metric-scale pose understanding. This is a true open world model, the foundation for embodied AI, robotics, autonomous systems, and hyper-realistic simulations that can run anywhere. Project: At our Zero-Human Company, we’re already running SANA-WM live in our core pipelines. It’s supercharging autonomous agent training, generating unlimited synthetic training data, and powering full end-to-end simulation loops, zero humans in the loop. The speed and control let us test thousands of edge-case scenarios overnight, iterate at lightspeed, and push our fully autonomous operations further than ever before. This is the kind of breakthrough that turns science fiction into daily reality. World models just leveled up — hard. The age of personal, local, controllable universes is here.
1:13

NVIDIA just unleashed SANA-WM and it’s an absolute MONSTER for the future of open source AI! A blazing-fast 2.6B-parameter open-source world model that doesn’t just generate video… it creates controllable, physics-rich, high-fidelity worlds on demand. Why this is insanely powerful: • One image + text prompt + 6-DoF camera trajectory → generates 720p videos up to 60 seconds long with buttery-smooth, precisely controlled camera movement. You’re not just watching, you’re piloting the simulation. • Runs locally on a single consumer GPU (RTX 5090 level) thanks to heavy distillation + NVFP4 quantization. Full 60-second clip denoised in ~34 seconds. No massive clusters required. • 36× higher throughput than previous open models while rivaling (or beating) closed industrial giants in visual quality and consistency. • Trained lightning-fast: ~213K public videos in just 15 days on 64 H100s. • Built with next-level tech: Hybrid Linear Attention, dual-branch camera control, two-stage pipeline, and rock-solid metric-scale pose understanding. This is a true open world model, the foundation for embodied AI, robotics, autonomous systems, and hyper-realistic simulations that can run anywhere. Project: At our Zero-Human Company, we’re already running SANA-WM live in our core pipelines. It’s supercharging autonomous agent training, generating unlimited synthetic training data, and powering full end-to-end simulation loops, zero humans in the loop. The speed and control let us test thousands of edge-case scenarios overnight, iterate at lightspeed, and push our fully autonomous operations further than ever before. This is the kind of breakthrough that turns science fiction into daily reality. World models just leveled up — hard. The age of personal, local, controllable universes is here.

Brian Roemmele

618,015 просмотров • 1 месяц назад

Cerebras inference is very fast. So fast that it changes how we think about configuring our LLMs for voice agent use cases. Kimi K2.6 is a 1T parameter reasoning model that Cerebras serves at 650 - 1,000 tokens per second (end-to-end throughput), with time to first token metrics as low as 150ms (latency). These numbers are two to three times faster than other similarly capable models. The biggest lever we get from this kind of speed is that we can use the model in reasoning mode, and still have excellent "time to first non-thinking token." This solves a big pain point we have in 2026 for voice agent use cases. Almost all recent innovation in post-training has focused on making models good at reasoning ("test time compute"). This is great, but it makes the user-facing model latency much, much slower. Which is a problem for conversational voice agents. We can run Kimi K2.6 with reasoning turned on, and get responses faster than other models produce with reasoning disabled. On my 30-turn voice agent benchmark, Kimi K2.6 with reasoning enabled ties GPT 5.1 and Haiku 4.5 with reasoning disabled, and is still about 200ms seconds faster! On my primary task agent benchmark, Kimi K2.6 is now the #2 model. It ranks just behind Gemini 3.5 Flash in "high" reasoning mode, and tied with GLM 5, Sonnet 4.6, and GPT 5.4 with reasoning set to "low." But Kimi K2.6 completes each turn in the agent loop in under 500ms. The other four models are all at least 3x slower. (Models only qualify for this benchmark if they can complete task turns at a P50 <4s.) A couple of other things that this speed buys us, for production voice agents: - Tool calls happen fast enough that we don't have to work around tool call latency in our pipeline design. - We can prompt the model to output structured data at the beginning of a response, followed by plain text for voice generation. This opens up possibilities like asking the model to do complex classification/generation tasks that influence the rest of the pipeline. For example, the model could create a detailed style prompt for a steerable TTS model, for each individual conversation turn. And, of course, you can use Kimi K2.6 with reasoning turned off. Cerebras calls this "instant" mode. Here's a video of a Cerebras Kimi K2.6 voice agent with voice-to-voice response time, measured at the client, under 500ms. This is the true response latency as perceived by the user, including all network and audio codec overhead, transcription and turn detection, Kimi K2.6 token generation, and voice generation. 500ms is, effectively, instant. So the Cerebras naming for this mode is a propos. :-)
0:40

Cerebras inference is very fast. So fast that it changes how we think about configuring our LLMs for voice agent use cases. Kimi K2.6 is a 1T parameter reasoning model that Cerebras serves at 650 - 1,000 tokens per second (end-to-end throughput), with time to first token metrics as low as 150ms (latency). These numbers are two to three times faster than other similarly capable models. The biggest lever we get from this kind of speed is that we can use the model in reasoning mode, and still have excellent "time to first non-thinking token." This solves a big pain point we have in 2026 for voice agent use cases. Almost all recent innovation in post-training has focused on making models good at reasoning ("test time compute"). This is great, but it makes the user-facing model latency much, much slower. Which is a problem for conversational voice agents. We can run Kimi K2.6 with reasoning turned on, and get responses faster than other models produce with reasoning disabled. On my 30-turn voice agent benchmark, Kimi K2.6 with reasoning enabled ties GPT 5.1 and Haiku 4.5 with reasoning disabled, and is still about 200ms seconds faster! On my primary task agent benchmark, Kimi K2.6 is now the #2 model. It ranks just behind Gemini 3.5 Flash in "high" reasoning mode, and tied with GLM 5, Sonnet 4.6, and GPT 5.4 with reasoning set to "low." But Kimi K2.6 completes each turn in the agent loop in under 500ms. The other four models are all at least 3x slower. (Models only qualify for this benchmark if they can complete task turns at a P50 <4s.) A couple of other things that this speed buys us, for production voice agents: - Tool calls happen fast enough that we don't have to work around tool call latency in our pipeline design. - We can prompt the model to output structured data at the beginning of a response, followed by plain text for voice generation. This opens up possibilities like asking the model to do complex classification/generation tasks that influence the rest of the pipeline. For example, the model could create a detailed style prompt for a steerable TTS model, for each individual conversation turn. And, of course, you can use Kimi K2.6 with reasoning turned off. Cerebras calls this "instant" mode. Here's a video of a Cerebras Kimi K2.6 voice agent with voice-to-voice response time, measured at the client, under 500ms. This is the true response latency as perceived by the user, including all network and audio codec overhead, transcription and turn detection, Kimi K2.6 token generation, and voice generation. 500ms is, effectively, instant. So the Cerebras naming for this mode is a propos. :-)

kwindla

40,319 просмотров • 1 месяц назад

I’ve been using GPT-5.6 Sol internally for the past two months, I've spent probably 25+ billion tokens. Here’s my review and comparison to Fable 5: > Let's start with the analogy because everyone seems to be giving theirs - GPT-5.6 is likely the last version of the GPT-5 training run series. It's kind of like an athlete at their peak. Through years of experience in the game, they've become the most reliable player and has the highest game IQ. But, there's no more room to grow. Fable on the other hand, being essentially the first version of a new training run, is the first round draft pick rookie. Raw talent mixed with the energy only a young person would have results in some incredible plays we didn't think possible, but also mistakes due to lack of experience. But that rookie will only improve and likely will be better than the veteran ever was because it's a new game and a new era. > GPT-5.6 is genuinely better at long, sustained work. With /goal, I've had it running complex projects for days with almost no intervention. It built a Minecraft-style game, kept adding features and mobs after the core game worked, and only stopped because I stopped the run. I never felt as though I had to jump in and guide it back to the right path. > It keeps finding useful work when you give it a concrete finish line. I had it recreate Excel with a loop. It inspected the real desktop excel app with Computer Use, comparing that against its own build, and closing the gaps. I stopped it after six days after it had built an incredible amount of functionality. > It's faster than other models in two different ways. The raw generation speed is higher, something OpenAI has been putting effort into. But it also takes a shorter path to solutions. It wanders less, changes less code, and generally knows how to get things done directly. In daily use, it feels about 2-3x times faster than Fable. That's my impression, not a controlled benchmark. The difference is large enough that I notice it constantly. > It works well across a wide range of tasks. I use it for one-line edits, quick questions, browser chores, and multi-day builds without changing my prompting style. Speaking of browser control, its the best ever I've used. To the point where I actually use it often. If a task lives on a website, GPT-5.6 usually opens the browser and does it there instead of asking for an API key or forcing everything through the terminal. When I switched back to GPT-5.5, it went straight to the command line even when the browser was clearly the better tool. > And it can handle real browser work, not just toy demos. During a data import, I had it monitor Supabase and resize instances as the load changed. It stayed on the dashboard, adjusted capacity, and checked the result without an API or a custom script. > I also gave it a full Google Workspace migration. It moved Forward Future from to preserved the old aliases, and configured MX, SPF, and DKIM. Before a consequential save, it stopped, explained exactly what would change, and waited for confirmation. > The reasoning setting matters a lot. Light is good for questions and small edits. High and Extra High are the sweet spots for serious work. Ultra usually takes longer than the extra thinking is worth and burns tokens. > I love that 5.6 is split into 3 sizes. Not only can you control speed and cost that way, but you still also have the thinking effort setting for each of them. Very precise controls. I just wish Codex automatically routed my prompts for me. > Its personality is blunt and a little bland. Claude feels warmer and more natural to talk to. GPT-5.6 is more clinical, but I like that for work. It gives me enough explanation and rarely pads the answer. I usually have to ask Fable to explain things more simply and/or more concise. > Its front-end taste has improved, but the default is predictable. Left alone, it turns websites into PowerPoint decks with huge statements and hard section breaks. The good news is that it takes design direction well and can revise without destroying the parts that already work. > It still makes confident mistakes. I asked it to rebuild parts of a system, and it told me the job was finished. Later, I found out it wasn't. Bits of its internal process also leak into the answer occasionally. > Claude Fable is more naturally autonomous on large, open-ended projects. GPT-5.6 is easier to reach for. I don't need to invent a huge project to justify using it. It works just as well for a small edit or browser chore. > GPT-5.6 is also cheaper. Sol costs $5 per million input tokens and $30 per million output tokens. Fable costs $10 and $50. Cached input is cheaper too. Still, cost per finished task matters more than cost per token. > GPT-5.6 isn't the best at everything, and it still needs supervision. But it generates faster, wanders less, works at almost any scale, and wastes less of my time. It's the model I have the most confidence in to get the job done right the first time. I put together a full breakdown with all the tests, prompts, and examples on a site. You can read it here:
7:46

I’ve been using GPT-5.6 Sol internally for the past two months, I've spent probably 25+ billion tokens. Here’s my review and comparison to Fable 5: > Let's start with the analogy because everyone seems to be giving theirs - GPT-5.6 is likely the last version of the GPT-5 training run series. It's kind of like an athlete at their peak. Through years of experience in the game, they've become the most reliable player and has the highest game IQ. But, there's no more room to grow. Fable on the other hand, being essentially the first version of a new training run, is the first round draft pick rookie. Raw talent mixed with the energy only a young person would have results in some incredible plays we didn't think possible, but also mistakes due to lack of experience. But that rookie will only improve and likely will be better than the veteran ever was because it's a new game and a new era. > GPT-5.6 is genuinely better at long, sustained work. With /goal, I've had it running complex projects for days with almost no intervention. It built a Minecraft-style game, kept adding features and mobs after the core game worked, and only stopped because I stopped the run. I never felt as though I had to jump in and guide it back to the right path. > It keeps finding useful work when you give it a concrete finish line. I had it recreate Excel with a loop. It inspected the real desktop excel app with Computer Use, comparing that against its own build, and closing the gaps. I stopped it after six days after it had built an incredible amount of functionality. > It's faster than other models in two different ways. The raw generation speed is higher, something OpenAI has been putting effort into. But it also takes a shorter path to solutions. It wanders less, changes less code, and generally knows how to get things done directly. In daily use, it feels about 2-3x times faster than Fable. That's my impression, not a controlled benchmark. The difference is large enough that I notice it constantly. > It works well across a wide range of tasks. I use it for one-line edits, quick questions, browser chores, and multi-day builds without changing my prompting style. Speaking of browser control, its the best ever I've used. To the point where I actually use it often. If a task lives on a website, GPT-5.6 usually opens the browser and does it there instead of asking for an API key or forcing everything through the terminal. When I switched back to GPT-5.5, it went straight to the command line even when the browser was clearly the better tool. > And it can handle real browser work, not just toy demos. During a data import, I had it monitor Supabase and resize instances as the load changed. It stayed on the dashboard, adjusted capacity, and checked the result without an API or a custom script. > I also gave it a full Google Workspace migration. It moved Forward Future from to preserved the old aliases, and configured MX, SPF, and DKIM. Before a consequential save, it stopped, explained exactly what would change, and waited for confirmation. > The reasoning setting matters a lot. Light is good for questions and small edits. High and Extra High are the sweet spots for serious work. Ultra usually takes longer than the extra thinking is worth and burns tokens. > I love that 5.6 is split into 3 sizes. Not only can you control speed and cost that way, but you still also have the thinking effort setting for each of them. Very precise controls. I just wish Codex automatically routed my prompts for me. > Its personality is blunt and a little bland. Claude feels warmer and more natural to talk to. GPT-5.6 is more clinical, but I like that for work. It gives me enough explanation and rarely pads the answer. I usually have to ask Fable to explain things more simply and/or more concise. > Its front-end taste has improved, but the default is predictable. Left alone, it turns websites into PowerPoint decks with huge statements and hard section breaks. The good news is that it takes design direction well and can revise without destroying the parts that already work. > It still makes confident mistakes. I asked it to rebuild parts of a system, and it told me the job was finished. Later, I found out it wasn't. Bits of its internal process also leak into the answer occasionally. > Claude Fable is more naturally autonomous on large, open-ended projects. GPT-5.6 is easier to reach for. I don't need to invent a huge project to justify using it. It works just as well for a small edit or browser chore. > GPT-5.6 is also cheaper. Sol costs $5 per million input tokens and $30 per million output tokens. Fable costs $10 and $50. Cached input is cheaper too. Still, cost per finished task matters more than cost per token. > GPT-5.6 isn't the best at everything, and it still needs supervision. But it generates faster, wanders less, works at almost any scale, and wastes less of my time. It's the model I have the most confidence in to get the job done right the first time. I put together a full breakdown with all the tests, prompts, and examples on a site. You can read it here:

Matthew Berman

183,440 просмотров • 5 дней назад

#Jasmy and #Janction are entering a new phase of expansion. The team aims to make its platform even more accessible, particularly by simplifying the development environment for application creators. A key focus is the introduction of an English interface, clearer documentation, and enhanced technical support. At the same time, #Jasmy is intensifying its international efforts, with particular attention to Southeast Asia. The year 2025 promises stronger communications and several major announcements ahead. Janction, on its part, has undergone a major transformation. It is no longer just a side project but now a true decentralized physical infrastructure built on a simple idea: everyone should be able to own and benefit from their own assets, including their data and computing power. Today, artificial intelligence, image generation, video rendering, and large-scale data processing all heavily rely on a single resource: the GPU. Originally designed for gaming, GPUs have become essential for any task that requires massive parallel processing. Unlike CPUs, GPUs can execute thousands of operations simultaneously, making them ideal for machine learning and AI models. However, this exponential demand has led to a global shortage. GPU prices are skyrocketing, lead times stretch up to a year, and the market is dominated by a few major players. In Japan and across Asia, the situation is especially strained. This is where Janction steps in with a disruptive approach. The idea is to allow any user to share an unused GPU, whether it’s in a gaming PC, a company server, a university lab, or even a cybercafé. In return, the owner gets paid. And to make this process smooth, simple, and secure, Janction relies on Docker technology. To visualize this, imagine a box containing everything needed to run an application, the code, libraries, and required files. Thanks to Docker, this box can be sent and run on any computer without conflicts or manual setup. This allows Janction to distribute AI or processing tasks across its network seamlessly. Each user receives a container, runs it via their GPU, and is paid automatically through smart contracts deployed on the network. The system is based on a fixed-rate subleasing model. Even if the GPU isn’t used 24/7, the owner still earns income. This is an ideal solution for schools, creative studios, researchers, or startups that have available resources but variable needs. Today, over 4,500 GPU nodes are already active in Japan, Hong Kong, and Singapore. The network offers fast block times and 99.9% reliability. The goal is ambitious: reach 100,000 nodes. To achieve this, Janction is targeting six main markets: AI startups, 3D and video studios, streaming platforms, research centers, game developers, and of course, owners of underutilized GPUs. At the same time, an Ethereum-based JANCTION token is in preparation. It will be used to reserve GPU power, participate in the ecosystem, and unlock additional rewards, including JASMY tokens. This dual-incentive system is designed to encourage the large-scale acquisition, sharing, and use of GPU power. The tokens will be tradable, storable, or reinvestable into hardware to further strengthen the network. #Janction’s strategy is clear, first, establish strong liquidity on recognized exchanges, then open access to a broad investor base, especially in #Japan, South Korea, and the United States.
3:23

#Jasmy and #Janction are entering a new phase of expansion. The team aims to make its platform even more accessible, particularly by simplifying the development environment for application creators. A key focus is the introduction of an English interface, clearer documentation, and enhanced technical support. At the same time, #Jasmy is intensifying its international efforts, with particular attention to Southeast Asia. The year 2025 promises stronger communications and several major announcements ahead. Janction, on its part, has undergone a major transformation. It is no longer just a side project but now a true decentralized physical infrastructure built on a simple idea: everyone should be able to own and benefit from their own assets, including their data and computing power. Today, artificial intelligence, image generation, video rendering, and large-scale data processing all heavily rely on a single resource: the GPU. Originally designed for gaming, GPUs have become essential for any task that requires massive parallel processing. Unlike CPUs, GPUs can execute thousands of operations simultaneously, making them ideal for machine learning and AI models. However, this exponential demand has led to a global shortage. GPU prices are skyrocketing, lead times stretch up to a year, and the market is dominated by a few major players. In Japan and across Asia, the situation is especially strained. This is where Janction steps in with a disruptive approach. The idea is to allow any user to share an unused GPU, whether it’s in a gaming PC, a company server, a university lab, or even a cybercafé. In return, the owner gets paid. And to make this process smooth, simple, and secure, Janction relies on Docker technology. To visualize this, imagine a box containing everything needed to run an application, the code, libraries, and required files. Thanks to Docker, this box can be sent and run on any computer without conflicts or manual setup. This allows Janction to distribute AI or processing tasks across its network seamlessly. Each user receives a container, runs it via their GPU, and is paid automatically through smart contracts deployed on the network. The system is based on a fixed-rate subleasing model. Even if the GPU isn’t used 24/7, the owner still earns income. This is an ideal solution for schools, creative studios, researchers, or startups that have available resources but variable needs. Today, over 4,500 GPU nodes are already active in Japan, Hong Kong, and Singapore. The network offers fast block times and 99.9% reliability. The goal is ambitious: reach 100,000 nodes. To achieve this, Janction is targeting six main markets: AI startups, 3D and video studios, streaming platforms, research centers, game developers, and of course, owners of underutilized GPUs. At the same time, an Ethereum-based JANCTION token is in preparation. It will be used to reserve GPU power, participate in the ecosystem, and unlock additional rewards, including JASMY tokens. This dual-incentive system is designed to encourage the large-scale acquisition, sharing, and use of GPU power. The tokens will be tradable, storable, or reinvestable into hardware to further strengthen the network. #Janction’s strategy is clear, first, establish strong liquidity on recognized exchanges, then open access to a broad investor base, especially in #Japan, South Korea, and the United States.

NeoXtrix

44,354 просмотров • 1 год назад

Last Friday on Pi Day, we held AI Dev 25, a new conference for AI Developers. Tickets had (unfortunately) sold out shortly after we announced their availability, but I came away energized by the day of coding and technical discussions with fellow AI Builders! Let me share here my observations from the event. I'd decided to start AI Dev because while there're great academic AI conferences that disseminate research work (such as NeurIPS, ICML and ICLR) and also great meetings held by individual companies, often focused on each company's product offerings, there were few vendor-neutral conferences for AI developers. With the wide range of AI tools now available, there is a rich set of opportunities for developers to build new things (and to share ideas on how to build things!), but also a need for a neutral forum that helps developers do so. Based on an informal poll, about half the attendees had traveled to San Francisco from outside the Bay Area for this meeting, including many who had come from overseas. I was thrilled by the enthusiasm to be part of this AI Builder community. To everyone who came, thank you! Other aspects of the event that struck me: - First, agentic AI continues to be a strong theme. The topic attendees most wanted to hear about (based on free text responses to our in-person survey at the start of the event) was agents! - Google's Paige Bailey talked about embedding AI in everything and using a wide range of models to do so. I also particularly enjoyed her demos of Astra and Deep Research agents. - Meta's Amit Sangani talked compellingly as usual about open models. Specifically, he described developers fine-tuning smaller models on specific data, resulting in superior performance than with large general purpose models. While there're still many companies using fine-tuning that should really just be prompting, I'm also seeing continued growth of fine-tuning in applications that are reaching scale and that are becoming valuable. - Many speakers also spoke about the importance of being pragmatic about what problems we are solving, as opposed to buying into the AGI hype. For example, Nebius' Roman Chernin put it simply: Focusing on solving real problems is important! - Lastly, I was excited to hear continued enthusiasm for the Voice Stack. Justin Uberti gave a talk about OpenAI’s realtime audio API to a packed room, with many people pulling out laptops to try things out themselves in code! has a strong “Learner First” mentality; our foremost goal is always to help learners. I was thrilled that a few attendees told me they enjoyed how technical the sessions were, and said they learned many things that they're sure they will use. (In fact, I, too, came away with a few ideas from the sessions!) I was also struck that, both during the talks and at the technical demo booths, the rooms were packed with attendees who were highly engaged throughout the whole day. I'm glad that we were able to have a meeting filled with technical and engineering discussions. I'm delighted that AI Dev 25 went off so well, and am grateful to all the attendees, volunteers, speakers, sponsors, partners, and team members that made the event possible. I regretted only that the physical size of the event space prevented us from admitting more attendees this time. There is something magical about bringing people together physically to share ideas, make friends, and to learn from and help each other. I hope we'll be able to bring even more people together in the future. [Original text: ]
3:20

Last Friday on Pi Day, we held AI Dev 25, a new conference for AI Developers. Tickets had (unfortunately) sold out shortly after we announced their availability, but I came away energized by the day of coding and technical discussions with fellow AI Builders! Let me share here my observations from the event. I'd decided to start AI Dev because while there're great academic AI conferences that disseminate research work (such as NeurIPS, ICML and ICLR) and also great meetings held by individual companies, often focused on each company's product offerings, there were few vendor-neutral conferences for AI developers. With the wide range of AI tools now available, there is a rich set of opportunities for developers to build new things (and to share ideas on how to build things!), but also a need for a neutral forum that helps developers do so. Based on an informal poll, about half the attendees had traveled to San Francisco from outside the Bay Area for this meeting, including many who had come from overseas. I was thrilled by the enthusiasm to be part of this AI Builder community. To everyone who came, thank you! Other aspects of the event that struck me: - First, agentic AI continues to be a strong theme. The topic attendees most wanted to hear about (based on free text responses to our in-person survey at the start of the event) was agents! - Google's Paige Bailey talked about embedding AI in everything and using a wide range of models to do so. I also particularly enjoyed her demos of Astra and Deep Research agents. - Meta's Amit Sangani talked compellingly as usual about open models. Specifically, he described developers fine-tuning smaller models on specific data, resulting in superior performance than with large general purpose models. While there're still many companies using fine-tuning that should really just be prompting, I'm also seeing continued growth of fine-tuning in applications that are reaching scale and that are becoming valuable. - Many speakers also spoke about the importance of being pragmatic about what problems we are solving, as opposed to buying into the AGI hype. For example, Nebius' Roman Chernin put it simply: Focusing on solving real problems is important! - Lastly, I was excited to hear continued enthusiasm for the Voice Stack. Justin Uberti gave a talk about OpenAI’s realtime audio API to a packed room, with many people pulling out laptops to try things out themselves in code! has a strong “Learner First” mentality; our foremost goal is always to help learners. I was thrilled that a few attendees told me they enjoyed how technical the sessions were, and said they learned many things that they're sure they will use. (In fact, I, too, came away with a few ideas from the sessions!) I was also struck that, both during the talks and at the technical demo booths, the rooms were packed with attendees who were highly engaged throughout the whole day. I'm glad that we were able to have a meeting filled with technical and engineering discussions. I'm delighted that AI Dev 25 went off so well, and am grateful to all the attendees, volunteers, speakers, sponsors, partners, and team members that made the event possible. I regretted only that the physical size of the event space prevented us from admitting more attendees this time. There is something magical about bringing people together physically to share ideas, make friends, and to learn from and help each other. I hope we'll be able to bring even more people together in the future. [Original text: ]

Andrew Ng

46,764 просмотров • 1 год назад

I know your timeline is flooded now with word salads of "insane, HER, 10 features you missed, we're so back". Sit down. Chill. Take a deep breath like Mark does in the demo . Let's think step by step: - Technique-wise, OpenAI has figured out a way to map audio to audio directly as first-class modality, and stream videos to a transformer in real-time. These require some new research on tokenization and architecture, but overall it's a data and system optimization problem (as most things are). High-quality data can come from at least 2 sources: 1) Naturally occurring dialogues on YouTube, podcasts, TV series, movies, etc. Whisper can be trained to identify speaker turns in a dialogue or separate overlapping speeches for automated annotation. 2) Synthetic data. Run the slow 3-stage pipeline using the most powerful models: speech1->text1 (ASR), text1->text2 (LLM), text2->speech2 (TTS). The middle LLM can decide when to stop and also simulate how to resume from interruption. It could output additional "thought traces" that are not verbalized to help generate better reply. Then GPT-4o distills directly from speech1->speech2, with optional auxiliary loss functions based on the 3-stage data. After distillation, these behaviors are now baked into the model without emitting intermediate texts. On the system side: the latency would not meet real-time threshold if every video frame is decompressed into an RGB image. OpenAI has likely developed their own neural-first, streaming video codec to transmit the motion deltas as tokens. The communication protocol and NN inference must be co-optimized. For example, there could be a small and energy-efficient NN running on the edge device that decides to transmit more tokens if the video is interesting, and fewer otherwise. - I didn't expect GPT-4o to be closer to GPT-5, the rumored "Arrakis" model that takes multimodal in and out. In fact, it's likely an early checkpoint of GPT-5 that hasn't finished training yet. The branding betrays a certain insecurity. Ahead of Google I/O, OpenAI would rather beat our mental projection of GPT-4.5 than disappoint by missing the sky-high expectation for GPT-5. A smart move to buy more time. - Notably, the assistant is much more lively and even a bit flirty. GPT-4o is trying (perhaps a bit too hard) to sound like HER. OpenAI is eating Character AI's lunch, with almost 100% overlap in form factor and huge distribution channels. It's a pivot towards more emotional AI with strong personality, which OpenAI seemed to actively suppress in the past. - Whoever wins Apple first wins big time. I see 3 levels of integration with iOS: 1) Ditch Siri. OpenAI distills a smaller-tier, purely on-device GPT-4o for iOS, with optional paid upgrade to use the cloud. 2) Native features to stream the camera or screen into the model. Chip-level support for neural audio/video codec. 3) Integrate with iOS system-level action API and smart home APIs. No one uses Siri Shortcuts, but it's time to resurrect. This could become the AI agent product with a billion users from the get-go. The FSD for smartphones with a Tesla-scale data flywheel.
14:22

I know your timeline is flooded now with word salads of "insane, HER, 10 features you missed, we're so back". Sit down. Chill. Take a deep breath like Mark does in the demo . Let's think step by step: - Technique-wise, OpenAI has figured out a way to map audio to audio directly as first-class modality, and stream videos to a transformer in real-time. These require some new research on tokenization and architecture, but overall it's a data and system optimization problem (as most things are). High-quality data can come from at least 2 sources: 1) Naturally occurring dialogues on YouTube, podcasts, TV series, movies, etc. Whisper can be trained to identify speaker turns in a dialogue or separate overlapping speeches for automated annotation. 2) Synthetic data. Run the slow 3-stage pipeline using the most powerful models: speech1->text1 (ASR), text1->text2 (LLM), text2->speech2 (TTS). The middle LLM can decide when to stop and also simulate how to resume from interruption. It could output additional "thought traces" that are not verbalized to help generate better reply. Then GPT-4o distills directly from speech1->speech2, with optional auxiliary loss functions based on the 3-stage data. After distillation, these behaviors are now baked into the model without emitting intermediate texts. On the system side: the latency would not meet real-time threshold if every video frame is decompressed into an RGB image. OpenAI has likely developed their own neural-first, streaming video codec to transmit the motion deltas as tokens. The communication protocol and NN inference must be co-optimized. For example, there could be a small and energy-efficient NN running on the edge device that decides to transmit more tokens if the video is interesting, and fewer otherwise. - I didn't expect GPT-4o to be closer to GPT-5, the rumored "Arrakis" model that takes multimodal in and out. In fact, it's likely an early checkpoint of GPT-5 that hasn't finished training yet. The branding betrays a certain insecurity. Ahead of Google I/O, OpenAI would rather beat our mental projection of GPT-4.5 than disappoint by missing the sky-high expectation for GPT-5. A smart move to buy more time. - Notably, the assistant is much more lively and even a bit flirty. GPT-4o is trying (perhaps a bit too hard) to sound like HER. OpenAI is eating Character AI's lunch, with almost 100% overlap in form factor and huge distribution channels. It's a pivot towards more emotional AI with strong personality, which OpenAI seemed to actively suppress in the past. - Whoever wins Apple first wins big time. I see 3 levels of integration with iOS: 1) Ditch Siri. OpenAI distills a smaller-tier, purely on-device GPT-4o for iOS, with optional paid upgrade to use the cloud. 2) Native features to stream the camera or screen into the model. Chip-level support for neural audio/video codec. 3) Integrate with iOS system-level action API and smart home APIs. No one uses Siri Shortcuts, but it's time to resurrect. This could become the AI agent product with a billion users from the get-go. The FSD for smartphones with a Tesla-scale data flywheel.

Jim Fan

991,628 просмотров • 2 лет назад

This Chinese mathematician earned $10,000 a month inventing the hardest problems to train Neural Networks through Scale AI. Today his income dropped to zero. All the solutions are now generated by the model itself. He used to just hold the problem in his head and spell it out in plain text. His work is pure intellect. An expert in higher mathematics, he made his money hand-crafting the trickiest puzzles to test and train neural networks via RLHF. The bastion of "human" logic rested entirely on him, on people with PhDs who knew how to invent the problem. The collapse is simple. The shift to RLAIF and synthetic data. The model plays against itself, builds trees of logical inference, and solves deeper than a human can even invent the problem. No PhD data engineers, no hand-written prompt-completion examples, no manual grading. Just the model, search algorithms, and Chain of Thought. Ready-made "smart human-time" still sells on the market for many times more. His old rate was $50–100 per problem. The internal "mini-app" was written by the model too. Inside there's no pretty shell, just bare logic with exact steps: input: the problem statement inference tree: thousands of branches per second check: every step verifies itself output: a proof a human never had time to invent And here is what the whole setup looked like. He no longer needs to write an example by hand. He gave the model a direct instruction in human words, without a single formal term: "solve the problem yourself and grade yourself yourself" That's it. After that the algorithm found the solution, checked it, and trained on its own result, with no human. → the contractor got $50–100 per problem written → from 5,000 to 10,000 a month → now that income is annulled → a query to a math LLM costs 1–5 cents → a quant or an actuary runs 150,000–250,000 a year → the margin for whoever packages this into an agent is nearly 100% In the author's own words: "I'm no longer able to invent a problem the machine can't solve. The examiner became dumber than the one he's examining." But honestly, he admits the crude mistake himself, and it's not in the math, it's in the positioning. He tied his income to selling "smart human-time", to crafting formulas by hand. As long as he sells formulas, he's left behind. The machine computes faster than he can invent the problem. He names the right move himself: the role shifts from "intellectual craftsman" to "systems architect." Then he doesn't sell his time, he manages compute, packaging that same LLM into an autonomous agent that runs 24/7. Out of everything I've seen this year about the disappearance of intellectual professions, this is the most honest example: $50 per problem zeroed out to 1 cent per query, a doctor of science losing to a search algorithm, one problem stated in human words instead of a hand-written dataset, and right away an out-loud admission of the wrong business model. The barrier to entry in higher mathematics just dropped to the level of "describe the task in words." The only question is who'll be the first to stop selling their time and start managing the machine's compute.
3:12

This Chinese mathematician earned $10,000 a month inventing the hardest problems to train Neural Networks through Scale AI. Today his income dropped to zero. All the solutions are now generated by the model itself. He used to just hold the problem in his head and spell it out in plain text. His work is pure intellect. An expert in higher mathematics, he made his money hand-crafting the trickiest puzzles to test and train neural networks via RLHF. The bastion of "human" logic rested entirely on him, on people with PhDs who knew how to invent the problem. The collapse is simple. The shift to RLAIF and synthetic data. The model plays against itself, builds trees of logical inference, and solves deeper than a human can even invent the problem. No PhD data engineers, no hand-written prompt-completion examples, no manual grading. Just the model, search algorithms, and Chain of Thought. Ready-made "smart human-time" still sells on the market for many times more. His old rate was $50–100 per problem. The internal "mini-app" was written by the model too. Inside there's no pretty shell, just bare logic with exact steps: input: the problem statement inference tree: thousands of branches per second check: every step verifies itself output: a proof a human never had time to invent And here is what the whole setup looked like. He no longer needs to write an example by hand. He gave the model a direct instruction in human words, without a single formal term: "solve the problem yourself and grade yourself yourself" That's it. After that the algorithm found the solution, checked it, and trained on its own result, with no human. → the contractor got $50–100 per problem written → from 5,000 to 10,000 a month → now that income is annulled → a query to a math LLM costs 1–5 cents → a quant or an actuary runs 150,000–250,000 a year → the margin for whoever packages this into an agent is nearly 100% In the author's own words: "I'm no longer able to invent a problem the machine can't solve. The examiner became dumber than the one he's examining." But honestly, he admits the crude mistake himself, and it's not in the math, it's in the positioning. He tied his income to selling "smart human-time", to crafting formulas by hand. As long as he sells formulas, he's left behind. The machine computes faster than he can invent the problem. He names the right move himself: the role shifts from "intellectual craftsman" to "systems architect." Then he doesn't sell his time, he manages compute, packaging that same LLM into an autonomous agent that runs 24/7. Out of everything I've seen this year about the disappearance of intellectual professions, this is the most honest example: $50 per problem zeroed out to 1 cent per query, a doctor of science losing to a search algorithm, one problem stated in human words instead of a hand-written dataset, and right away an out-loud admission of the wrong business model. The barrier to entry in higher mathematics just dropped to the level of "describe the task in words." The only question is who'll be the first to stop selling their time and start managing the machine's compute.

Blaze

49,109 просмотров • 1 месяц назад

A new 30-minute presentation from Ashok Elluswamy, Tesla’s VP of AI, has been released, where he talks about FSD, AI and the team’s latest progress. Highlight from the presentation: • Tesla's vehicle fleet can provide 500 years of driving data every single day. Curse of Dimensionality: • 8 cameras at high frame rate = billions of tokens per 30 seconds of driving context. • Tesla must compress and extract the right correlations between sensory input and control actions. Data Advantage: • Tesla has access to a “Niagara Falls of data” — hundreds of years’ worth of collective fleet driving. • Uses smart data triggers to capture rare corner cases (e.g., complex intersections, unpredictable behavior). Quality and Efficiency: • Extracts only the essential data needed to train models efficiently. Debugging and Interpretability: • Even though the system is end-to-end, Tesla can still prompt the model to output interpretable data: 3D occupancy, road boundaries, objects, signs, traffic lights, etc. • Natural language querying: ask the model why it made a certain decision. • These auxiliary predictions don’t drive the car but help engineers debug and ensure safety. Tesla’s Advanced Gaussian Splatting (3D Scene Modeling): • Tesla developed a custom, ultra-fast Gaussian splatting system to reconstruct 3D scenes from limited camera views. • Produces crisp, accurate 3D renderings even from few camera angles — far better than standard NeRF/splatting approaches. • Enables rapid visual debugging of the driving environment in 3D. Evaluation & World Models: • Evaluation is the hardest challenge: models may perform well offline but fail in real-world conditions. • Tesla builds balanced, diverse evaluation datasets focusing on edge cases — not just easy highway driving. Introduced a learned world simulator (neural network-generated video engine): • Can simulate 8 Tesla camera feeds simultaneously — fully synthetic. • Used for testing, training, and reinforcement learning. • Allows adversarial event injection (e.g., adding a pedestrian or vehicle cutting in). • Enables replaying past failures to verify new model improvements. • Can run in near real-time, letting testers “drive” inside a simulated world. What’s Next: • Scale robotaxi service globally. • Unlock full autonomy across the entire Tesla fleet. • Cybercab: next-gen 2-seat vehicle designed specifically for robotaxi use, targeting lowest transportation cost (cheaper than public transit). • Same neural networks will power Optimus humanoid robot. • The same video generation system is now being applied to Optimus. • The system can simulate and plan movement for robots, adapting easily to new forms. via the International Conference on Computer Vision (ICCV). Full presentation:
0:29

A new 30-minute presentation from Ashok Elluswamy, Tesla’s VP of AI, has been released, where he talks about FSD, AI and the team’s latest progress. Highlight from the presentation: • Tesla's vehicle fleet can provide 500 years of driving data every single day. Curse of Dimensionality: • 8 cameras at high frame rate = billions of tokens per 30 seconds of driving context. • Tesla must compress and extract the right correlations between sensory input and control actions. Data Advantage: • Tesla has access to a “Niagara Falls of data” — hundreds of years’ worth of collective fleet driving. • Uses smart data triggers to capture rare corner cases (e.g., complex intersections, unpredictable behavior). Quality and Efficiency: • Extracts only the essential data needed to train models efficiently. Debugging and Interpretability: • Even though the system is end-to-end, Tesla can still prompt the model to output interpretable data: 3D occupancy, road boundaries, objects, signs, traffic lights, etc. • Natural language querying: ask the model why it made a certain decision. • These auxiliary predictions don’t drive the car but help engineers debug and ensure safety. Tesla’s Advanced Gaussian Splatting (3D Scene Modeling): • Tesla developed a custom, ultra-fast Gaussian splatting system to reconstruct 3D scenes from limited camera views. • Produces crisp, accurate 3D renderings even from few camera angles — far better than standard NeRF/splatting approaches. • Enables rapid visual debugging of the driving environment in 3D. Evaluation & World Models: • Evaluation is the hardest challenge: models may perform well offline but fail in real-world conditions. • Tesla builds balanced, diverse evaluation datasets focusing on edge cases — not just easy highway driving. Introduced a learned world simulator (neural network-generated video engine): • Can simulate 8 Tesla camera feeds simultaneously — fully synthetic. • Used for testing, training, and reinforcement learning. • Allows adversarial event injection (e.g., adding a pedestrian or vehicle cutting in). • Enables replaying past failures to verify new model improvements. • Can run in near real-time, letting testers “drive” inside a simulated world. What’s Next: • Scale robotaxi service globally. • Unlock full autonomy across the entire Tesla fleet. • Cybercab: next-gen 2-seat vehicle designed specifically for robotaxi use, targeting lowest transportation cost (cheaper than public transit). • Same neural networks will power Optimus humanoid robot. • The same video generation system is now being applied to Optimus. • The system can simulate and plan movement for robots, adapting easily to new forms. via the International Conference on Computer Vision (ICCV). Full presentation:

Sawyer Merritt

1,286,614 просмотров • 8 месяцев назад