正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

1/ First, how do you benchmark an AI model? You give different models the same test cases, and score them. You can do the same with voice models using Word Error Rate (WER). You generate the audio, transcribe it with a speech recognition (ASR) model, and compare that transcript... show more

Cartesia

15,571 subscribers

10,184 次观看 • 23 天前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

7 条评论

Cartesia 的头像

Cartesia23 天前

We're rigorous about model evals at Cartesia because there's no shortcut. Quality is multidimensional, and no single number captures it. Here’s why benchmarking TTS is extremely difficult ⬇

Cartesia 的头像

Cartesia23 天前

2/ WER has another problem. The score depends on the ASR model you use to transcribe the audio. If the ASR struggles with an accent, language or domain-specific word, the TTS model gets penalized for the ASR's mistake. The inverse can happen too. A strong ASR can infer unclear speech and make a bad generation look better than it sounds.

Cartesia 的头像

Cartesia23 天前

3/ Which means we should also measure whether the audio actually sounds good. One way to do that at scale is to get another model to listen to the audio and predict its quality. But these automated quality scores have blind (deaf) spots too. In one study, researchers added increasingly wrong Japanese pitch accents to generated speech. Human ratings dropped from 4.00 to 2.16. But the automated scores didn’t move, or even went up in some cases.

Cartesia 的头像

Cartesia23 天前

4/ Which is why human evals are still the gold standard. Artificial Analysis, for example, gets people to listen to two samples and choose which one they prefer. Those preferences are then used to rank the models. But human scores aren't completely stable. Results can change based on who's listening, the instructions they're given, headphones vs speakers, volume, or even how tired they are.

Cartesia 的头像

Cartesia23 天前

5/ Then there's what you actually test. Most TTS evals use short clips because they're easier to generate, listen to and score. But a model can sound great for a few seconds and start breaking over a longer conversation. The speaker can drift, or the accent and pace can change. And when speech is streamed in real time, you get completely different failures like awkward joins, repeated words or cut-off sounds.

Cartesia 的头像

Cartesia23 天前

6/ A model can also perform really well on generic speech and fall apart on medical terms, legal text, chemistry, gaming or customer support. Each domain has different expectations around pronunciation, pacing and what’s correct. And gets harder across languages, where what counts as an “error” can change by language and locale. So being good on one benchmark doesn't necessarily mean being good on your use case.

Cartesia 的头像

Cartesia23 天前

7/ So don't just read the leaderboard. Listen for yourself. Test the model for the language, context and product you're actually building. That being said, we're still #1 though :)

相关视频

Which AI model actually makes the better image? Try the Model Arena on BudgetPixel AI — use the same prompt, compare different models side by side, and vote for your favorite result. You can also earn extra credits while participating. No specs. No hype. Just the outputs.

Which AI model actually makes the better image? Try the Model Arena on BudgetPixel AI — use the same prompt, compare different models side by side, and vote for your favorite result. You can also earn extra credits while participating. No specs. No hype. Just the outputs.

BudgetPixel AI

51,380 次观看 • 25 天前

a Google researcher walked into MIT and made an AI do math correctly by adding seven words to the prompt. the seven words: "you are an MIT mathematician." drop them, model gets it wrong. add them, right. same model. same question. every time. Carter Smith. runs Gemini at Google. 1 hour. free. he then spent the next 50 minutes explaining why. it is the cleanest hour on how LLMs actually work I have seen in two years. you will come back to this. save it now.

a Google researcher walked into MIT and made an AI do math correctly by adding seven words to the prompt. the seven words: "you are an MIT mathematician." drop them, model gets it wrong. add them, right. same model. same question. every time. Carter Smith. runs Gemini at Google. 1 hour. free. he then spent the next 50 minutes explaining why. it is the cleanest hour on how LLMs actually work I have seen in two years. you will come back to this. save it now.

Raytar

20,399 次观看 • 4 个月前

With the launch of GLM 5.2 this week, I see everyone asking "have open models caught up to closed models?" The more interesting question that's getting missed: what can you do with an open model that you can't do with a closed one? You can specialize them. And when you do, the number of economically valuable tasks open models can do actually subsumes that of closed models. Charlie O'Neill explaining this:

With the launch of GLM 5.2 this week, I see everyone asking "have open models caught up to closed models?" The more interesting question that's getting missed: what can you do with an open model that you can't do with a closed one? You can specialize them. And when you do, the number of economically valuable tasks open models can do actually subsumes that of closed models. Charlie O'Neill explaining this:

Madison Kanna

18,149 次观看 • 3 个月前

Text-only models now have vision in Command Code. • A vision model reads the attached image • Your main model gets the context in the same thread • /config lets you choose the vision model Stop describing screenshots. Just attach them.

Text-only models now have vision in Command Code. • A vision model reads the attached image • Your main model gets the context in the same thread • /config lets you choose the vision model Stop describing screenshots. Just attach them.

Command Code

19,556 次观看 • 1 个月前

Choose a model (any model) and build your application with it. Do not spend time swapping models early on. Do not try to optimize before you have a working system. This is one of the first recommendations I make to every new team I consult with. Eventually, it will be time to optimize the model. • You may need a cheaper model • You may need a faster model • You might need a smarter model Good luck if you stitched together 12 different APIs and SDKs from 7 different vendors. Over half of the companies I consult for run on Microsoft software and have access to Microsoft Foundry. Microsoft Foundry is a complete agentic ecosystem. If you're in that world and building AI applications, Microsoft Foundry is where everything lives: • Models (largest selection in the market) • Agentic SDK (Python, C#, JavaScript/TypeScript) • Tools • Evaluations • Monitoring They are fully integrated with GitHub and Visual Studio Code. The best part: Their agentic platform is fully agnostic of the models you use. You can integrate with any model using the same OpenAI-style API. Swapping one model for another takes 1 second.

Choose a model (any model) and build your application with it. Do not spend time swapping models early on. Do not try to optimize before you have a working system. This is one of the first recommendations I make to every new team I consult with. Eventually, it will be time to optimize the model. • You may need a cheaper model • You may need a faster model • You might need a smarter model Good luck if you stitched together 12 different APIs and SDKs from 7 different vendors. Over half of the companies I consult for run on Microsoft software and have access to Microsoft Foundry. Microsoft Foundry is a complete agentic ecosystem. If you're in that world and building AI applications, Microsoft Foundry is where everything lives: • Models (largest selection in the market) • Agentic SDK (Python, C#, JavaScript/TypeScript) • Tools • Evaluations • Monitoring They are fully integrated with GitHub and Visual Studio Code. The best part: Their agentic platform is fully agnostic of the models you use. You can integrate with any model using the same OpenAI-style API. Swapping one model for another takes 1 second.

Santiago

12,014 次观看 • 7 个月前

You can now try Llama 3.1 405B for free (link below)! This is the largest open-source model out there, and for the first time, an open model is competitive with closed models. This time around, Meta did something new: Llama 3.1 has a license that allows developers to use it to enhance other models. For the first time, you can distill Llama 3.1 405B's capabilities into a smaller, more practical model for your use case. First, here is the link where you can play with Llama 3.1 for free: The model is hosted in Tune Studio, an end-to-end platform for developing applications using Large Language Models. They are sponsoring this post. Take a look at the attached video. It will show you how you can fine-tune a simple model using Llama 3.1 without leaving the platform: 1. You can create an empty dataset 2. Use the playground to generate and record interactions with Llama 3.1 3. Modify the dataset directly using the playground 4. Export the data and fine-tune a smaller model Fast and easy! As long as you have a web browser, you can start experimenting with fine-tuning and Llama 3.1. That's all it takes!

You can now try Llama 3.1 405B for free (link below)! This is the largest open-source model out there, and for the first time, an open model is competitive with closed models. This time around, Meta did something new: Llama 3.1 has a license that allows developers to use it to enhance other models. For the first time, you can distill Llama 3.1 405B's capabilities into a smaller, more practical model for your use case. First, here is the link where you can play with Llama 3.1 for free: The model is hosted in Tune Studio, an end-to-end platform for developing applications using Large Language Models. They are sponsoring this post. Take a look at the attached video. It will show you how you can fine-tune a simple model using Llama 3.1 without leaving the platform: 1. You can create an empty dataset 2. Use the playground to generate and record interactions with Llama 3.1 3. Modify the dataset directly using the playground 4. Export the data and fine-tune a smaller model Fast and easy! As long as you have a web browser, you can start experimenting with fine-tuning and Llama 3.1. That's all it takes!

Santiago

55,609 次观看 • 2 年前

OpenAI researcher David Holtz reveals workers who don't adapt their prompting to newer models leave 50% of the alpha on the table because half of AI's value comes from how you interact with it: "We ran a lab experiment. We give people the same task to do, but we give them models of different capabilities. How much of the improvement when we give people more capable models comes from the fact that we gave them a better model vs the fact that they actually update and adapt the way they prompt and interact with these models?" "For tasks where precision matters, where you have some output you're trying to get and you're trying to steer the model, which is different from a creative task like brainstorming." "For these steering-based tasks, when you give people a better model, about 50% of the improvement we see in how well they're able to complete the task comes from changes in how they interact with the model." "If you just kept the way you interact with the model static and just made the capabilities better and better, you'd leave 50% of the alpha basically on the table." OpenAI

OpenAI researcher David Holtz reveals workers who don't adapt their prompting to newer models leave 50% of the alpha on the table because half of AI's value comes from how you interact with it: "We ran a lab experiment. We give people the same task to do, but we give them models of different capabilities. How much of the improvement when we give people more capable models comes from the fact that we gave them a better model vs the fact that they actually update and adapt the way they prompt and interact with these models?" "For tasks where precision matters, where you have some output you're trying to get and you're trying to steer the model, which is different from a creative task like brainstorming." "For these steering-based tasks, when you give people a better model, about 50% of the improvement we see in how well they're able to complete the task comes from changes in how they interact with the model." "If you just kept the way you interact with the model static and just made the capabilities better and better, you'd leave 50% of the alpha basically on the table." OpenAI

MTS

15,590 次观看 • 1 个月前

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏

Lior Alexander

231,729 次观看 • 3 年前

Tired of guessing which AI model is best? 🤔 We've got you covered! 👀 Here's a sneak peek of our new model comparison feature in action! Run the same prompt across multiple models and see which one works best for YOU. Which models are you excited to test? #Inferium #AI #Inference #Compare

Tired of guessing which AI model is best? 🤔 We've got you covered! 👀 Here's a sneak peek of our new model comparison feature in action! Run the same prompt across multiple models and see which one works best for YOU. Which models are you excited to test? #Inferium #AI #Inference #Compare

Inferium

12,362 次观看 • 1 年前

How to use 50+ API keys (models) for FREE on OpenClaw API??? - go to - login or register your account - click on "more models" - click on "use case" and select what you need it for - choose the model and open it - click on "view code" → "Generate API key" many models don't allow direct deploy, so use the "view code" button to generate API access basically Nvidia NIM gives you the ability to test almost any model from their list for FREE some of them are not worse than GPT 5.2 or Claude Opus 4.6, some might even perform better depending on the task how to understand if a model is efficient and compare it with others??? - go to - type the model name in search - click on "benchmarks" - you’ll see performance tests and rankings this way you can easily compare free models with paid ones of course there are RPM limits, on many models it’s around ~40 requests per minute each model is different, after generating the API key, RPM limits are shown in the top-right corner nothing stops you from using them, many work perfectly fine, super solid option for first tests and for learning OpenClaw or any other system where you need an AI API model

How to use 50+ API keys (models) for FREE on OpenClaw API??? - go to - login or register your account - click on "more models" - click on "use case" and select what you need it for - choose the model and open it - click on "view code" → "Generate API key" many models don't allow direct deploy, so use the "view code" button to generate API access basically Nvidia NIM gives you the ability to test almost any model from their list for FREE some of them are not worse than GPT 5.2 or Claude Opus 4.6, some might even perform better depending on the task how to understand if a model is efficient and compare it with others??? - go to - type the model name in search - click on "benchmarks" - you’ll see performance tests and rankings this way you can easily compare free models with paid ones of course there are RPM limits, on many models it’s around ~40 requests per minute each model is different, after generating the API key, RPM limits are shown in the top-right corner nothing stops you from using them, many work perfectly fine, super solid option for first tests and for learning OpenClaw or any other system where you need an AI API model

Ronin

58,974 次观看 • 7 个月前

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Santiago

164,162 次观看 • 2 年前

Pocket TTS goes multilingual! Now you can use our 100M-parameter models to generate speech in six languages, fast enough to run real-time without a GPU. We also improved the quality of the English model while keeping the same size. And all of this is open-source.

Pocket TTS goes multilingual! Now you can use our 100M-parameter models to generate speech in six languages, fast enough to run real-time without a GPU. We also improved the quality of the English model while keeping the same size. And all of this is open-source.

kyutai

30,012 次观看 • 5 个月前

You can attend a service and leave exactly the same person who walked in. Or you can have an encounter. And when you have an encounter with God — you don't walk out the same. You don't talk the same. You don't see the same. You don't carry the same things you carried when you walked through the door.

You can attend a service and leave exactly the same person who walked in. Or you can have an encounter. And when you have an encounter with God — you don't walk out the same. You don't talk the same. You don't see the same. You don't carry the same things you carried when you walked through the door.

Paula White-Cain

12,355 次观看 • 3 个月前

I'm willing to die on this hill: The best voice models are those with the best accuracy with key entities, not those that optimize for WER (Word Error Rate). Most speech-to-text providers optimize for WER, but in production applications, WER is not that relevant. Getting 95% of the words right is useless if you miss the customer's name, their phone number, or the street address they just spelled out letter by letter. The team at Gladia ran a very cool benchmark: • 1,000+ call center conversations • Lots of background noise • Focus on extracting names, phone numbers, addresses, locations, etc. The Gladia model outperformed every other state-of-the-art model by up to 17%! This is exactly the data that matters to companies using these models. You get this wrong, and everything downstream breaks. A few other things worth mentioning: • Latency on partials: < 150ms • 100+ languages supported • Dynamic language detection • Overall WER at 5.97% Definitely worth checking for anyone using voice models: Thanks to the Gladia team for collaborating with me on this post.

I'm willing to die on this hill: The best voice models are those with the best accuracy with key entities, not those that optimize for WER (Word Error Rate). Most speech-to-text providers optimize for WER, but in production applications, WER is not that relevant. Getting 95% of the words right is useless if you miss the customer's name, their phone number, or the street address they just spelled out letter by letter. The team at Gladia ran a very cool benchmark: • 1,000+ call center conversations • Lots of background noise • Focus on extracting names, phone numbers, addresses, locations, etc. The Gladia model outperformed every other state-of-the-art model by up to 17%! This is exactly the data that matters to companies using these models. You get this wrong, and everything downstream breaks. A few other things worth mentioning: • Latency on partials: < 150ms • 100+ languages supported • Dynamic language detection • Overall WER at 5.97% Definitely worth checking for anyone using voice models: Thanks to the Gladia team for collaborating with me on this post.

Santiago

25,204 次观看 • 8 个月前

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.

Santiago

384,573 次观看 • 3 年前

Your AI sucks. And it's not the model. It's you. "Make it better" isn't a prompt, it's a guessing game. I changed one step in my process and the output quality completely flipped. Same ideas, same models, totally different results.

Your AI sucks. And it's not the model. It's you. "Make it better" isn't a prompt, it's a guessing game. I changed one step in my process and the output quality completely flipped. Same ideas, same models, totally different results.

Flick

322,001 次观看 • 2 个月前

Erik Voorhees explains how you can combine different AI tools to beat any single model "We're working on a concept called Minds. If you put modules of AI systems together the right way, you can get results better than any single model itself" "All the labs are focused on their models and beating each other's scores, but if you combine these things in creative interesting ways you get stuff no model can do on its own" "If no new models came out for five years, the advances in AI would still continue apace because people have not realized how much they can do even with the stuff created two years ago"

Erik Voorhees explains how you can combine different AI tools to beat any single model "We're working on a concept called Minds. If you put modules of AI systems together the right way, you can get results better than any single model itself" "All the labs are focused on their models and beating each other's scores, but if you combine these things in creative interesting ways you get stuff no model can do on its own" "If no new models came out for five years, the advances in AI would still continue apace because people have not realized how much they can do even with the stuff created two years ago"

Market Bubble

16,864 次观看 • 4 个月前