Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introducing CADGenBench: measure how well AI systems produce engineering-grade 3D parts! While current models can generate 3D parts, they are far from precise enough to build functional parts. We built a benchmark to systematically measure their capabilities on two tasks: 1. Generation from an engineering drawing of a part... 2. Editing: given an existing STEP file and a requested change The benchmark is tool-agnostic. It makes no assumptions about how you build the model. You can vary the LLM, and you can vary the environment. Use build123d, Onshape, Autodesk, or a model without an LLM entirely. We open sourced the scoring engine and a reference baseline on top of build123d. A collaboration between Hugging Face and Mecado! Submission space: Code repository:show more

Michael Rabinovich

1,665 subscribers

65,766 views • 4 months ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

9 Comments

Mecado4 months ago

Can LLMs understand a drawing? Can they infer design intent? (The answer is... maybe not yet)

cemkaya4 months ago

@earthtojake can you submit text-2-cad to this ?

Jake Dahn4 months ago

thank you for sharing all of this!

Dan Connolly4 months ago

This is cool! @bsk1087

MakerMatters?4 months ago

Much appreciated Michael, this will prove super useful in getting cadgenie on webcad better o7.

GoFly4 months ago

对 AI模型生成 CAD 能力评测的标准嘛？

Reindustrialize Who's Who4 months ago

Important work!

Colin4 months ago

@grok what are some clever use cases for this

Morgan McGuire4 months ago

Nice one @MikushRab!

Related Videos

New open-source 3D world-generation model. I'm rendering a couple of worlds in the video, so check it out. You'll find the GitHub and the Hugging Face links to the model below. This is a multi-modal world model that you can use for a bunch of things: • To generate new worlds • To reconstruct worlds • To simulate 3D interactive worlds from a prompt, images, or a video You can edit the 3D outputs in Unity and Unreal Engine (they export as meshes, 3DGS files, and point clouds). You can also generate 3D characters in the world and walk around. Pretty fun stuff!

New open-source 3D world-generation model. I'm rendering a couple of worlds in the video, so check it out. You'll find the GitHub and the Hugging Face links to the model below. This is a multi-modal world model that you can use for a bunch of things: • To generate new worlds • To reconstruct worlds • To simulate 3D interactive worlds from a prompt, images, or a video You can edit the 3D outputs in Unity and Unreal Engine (they export as meshes, 3DGS files, and point clouds). You can also generate 3D characters in the world and walk around. Pretty fun stuff!

Santiago

65,446 views • 6 months ago

Yann LeCun (Yann LeCun) just revealed why he left Meta, Why are LLMs an extremely narrow field, and how "World model" is the way to build really meaningful agentic future. 🎯 Beautiful and simple in 1.5 mints. "I can’t imagine we can build agentic systems without those systems having the ability to predict, in advance, what the consequences of their actions are going to be. The way we act in the world is that we can predict the consequences of our actions, and that’s what allows us to plan. So what is a world model? Given the state of an environment, a system you want to control at time t, and given an action or intervention you imagine taking, can you predict the state of the world (or the system) at time t + 1? If you can, that’s a world model. You don’t do this at a pixel level, if it’s video. You do this in an abstract representation space, and that’s a crucial key insight." --- From 'AI House Davos" YT channel (full link in comment)

Yann LeCun (Yann LeCun) just revealed why he left Meta, Why are LLMs an extremely narrow field, and how "World model" is the way to build really meaningful agentic future. 🎯 Beautiful and simple in 1.5 mints. "I can’t imagine we can build agentic systems without those systems having the ability to predict, in advance, what the consequences of their actions are going to be. The way we act in the world is that we can predict the consequences of our actions, and that’s what allows us to plan. So what is a world model? Given the state of an environment, a system you want to control at time t, and given an action or intervention you imagine taking, can you predict the state of the world (or the system) at time t + 1? If you can, that’s a world model. You don’t do this at a pixel level, if it’s video. You do this in an abstract representation space, and that’s a crucial key insight." --- From 'AI House Davos" YT channel (full link in comment)

Rohan Paul

383,741 views • 8 months ago

Dreamina Seedance 2.0 is by far the best video model I've tried, and Dreamina makes it even better. You can try it in the link below. It's not just about the quality of the output, but about how much control you have over your video's look. Watch my video here. I'm attaching reference images and asking the model to generate a video using them. I can reference each image using the @ symbol to instruct the model which image to use. You can even upload a clip and use its camera movement, styles from an image, and audio vibe from a track. By the way, you can take an existing video and replace, remove, or add elements to it while the model preserves everything else. This is the closest we've gotten to "editing videos like photos".

Dreamina Seedance 2.0 is by far the best video model I've tried, and Dreamina makes it even better. You can try it in the link below. It's not just about the quality of the output, but about how much control you have over your video's look. Watch my video here. I'm attaching reference images and asking the model to generate a video using them. I can reference each image using the @ symbol to instruct the model which image to use. You can even upload a clip and use its camera movement, styles from an image, and audio vibe from a track. By the way, you can take an existing video and replace, remove, or add elements to it while the model preserves everything else. This is the closest we've gotten to "editing videos like photos".

Santiago

46,622 views • 6 months ago

Building with AI gets easier every day. Here is an open-source library that makes integrating AI into an application extremely easy: Star the repository! This library alone can make React the best front-end framework out there! There are a bunch of cool things I like about CopilotKit. Here are 3 of them: 1. It allows you to take any -powered agent and bring it into your application. (This is a brand-new feature!) 2. You can build an AI-powered chatbot in your application. The chatbot will have access to your context and can act on the application. 3. You can build a RAG workflow to process and answer questions from a real-time knowledge base. I recorded a video to show you how simple it is to make some of this happen. A few lines of code, and you are in business. Here is a link to the sample application: CopilotKit is open-source. You can self-host it. You can use it with any LLM. Thanks to the team for showing me their tool and collaborating with me on this post!

Santiago

108,884 views • 2 years ago

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Santiago

164,162 views • 2 years ago

AVERI standards director Patricia Paskov reveals how they benchmarked Gemini without Google DeepMind ever seeing the benchmark or the benchmark owner ever seeing the model: "This is building on a pilot that was done at UK AISI with Anthropic two years ago, with a toy model and a toy benchmark. OpenMined provided the secure enclave in which both went. Neither side saw the other's." "What we did is we built on this, but using a Gemini model. We used a frontier model, and we also used a real benchmark from MLCommons. There are the proper firewalls in place such that these are both going into the secure enclave that OpenMined makes." "It's privacy preserving, and it gives the guarantees of privacy and security to both sides such that you can run the evaluation and spit out the results." "I'm really excited about leveraging stronger engineering tools as solutions to constraints like privacy and security that can unlock opportunities in auditing." AVERI

AVERI standards director Patricia Paskov reveals how they benchmarked Gemini without Google DeepMind ever seeing the benchmark or the benchmark owner ever seeing the model: "This is building on a pilot that was done at UK AISI with Anthropic two years ago, with a toy model and a toy benchmark. OpenMined provided the secure enclave in which both went. Neither side saw the other's." "What we did is we built on this, but using a Gemini model. We used a frontier model, and we also used a real benchmark from MLCommons. There are the proper firewalls in place such that these are both going into the secure enclave that OpenMined makes." "It's privacy preserving, and it gives the guarantees of privacy and security to both sides such that you can run the evaluation and spit out the results." "I'm really excited about leveraging stronger engineering tools as solutions to constraints like privacy and security that can unlock opportunities in auditing." AVERI

MTS

14,540 views • 15 days ago

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

Just found an open-source tool that lets you see how well an AI can actually use a computer. it’s called OSWorld 2.0. you give the AI a real task on a computer, then watch to see if it can figure everything out and finish it. it gives you: → real computer tasks, not simple questions → tasks that can take many steps to finish → the same computer setup for every AI → websites made for testing → all the files and tasks for free this is also the environment used to test computer-use systems from OpenAI and Anthropic. so if you’re building AI agents, researching computer use, or just want to see what these models can actually do, you can run the same tests yourself instead of relying only on benchmark scores. the labs publish the scores. this repo is where the actual tests come from. bookmark this.

MIKE

14,387 views • 1 month ago

Open-weight AI models are getting seriously capable. IQuest-Q1 just dropped, and it’s built for more than generating code. → ~320B total parameters, ~15B active per token → Strong results on NL2Repo and CyberGym → Competitive on Terminal-Bench 2.1 and Agents’ Last Exam → Can build interactive 3D apps and games from natural language → Can use tools, inspect logs, modify code, and verify fixes → Handles long-horizon, multi-step tasks across real work environments The wildest part? IQuest-Q1 helps diagnose capabilities, design training steps, and execute parts of R&D. Open weights. Open research. This feels less like “another LLM” and more like a glimpse at AI that can actually do the work. Technical Blog： Hugging Face： GitHub：

Open-weight AI models are getting seriously capable. IQuest-Q1 just dropped, and it’s built for more than generating code. → ~320B total parameters, ~15B active per token → Strong results on NL2Repo and CyberGym → Competitive on Terminal-Bench 2.1 and Agents’ Last Exam → Can build interactive 3D apps and games from natural language → Can use tools, inspect logs, modify code, and verify fixes → Handles long-horizon, multi-step tasks across real work environments The wildest part? IQuest-Q1 helps diagnose capabilities, design training steps, and execute parts of R&D. Open weights. Open research. This feels less like “another LLM” and more like a glimpse at AI that can actually do the work. Technical Blog： Hugging Face： GitHub：

Atul Kumar

79,465 views • 15 days ago

I gave an AI one instruction: “Build me a playable 3D game from scratch.” Then I let it cook. 👀 I tested Tencent Hy4 preview inside WorkBuddy, and it went way beyond just generating a few snippets of code. It could: → Plan the game structure → Generate the code across files → Create the UI and interactions → Debug problems along the way → Iterate on the project while keeping context That last part is what caught my attention. Hy4 preview is built for these long-horizon tasks, with 770B total parameters, 49B active parameters, and a 1M-token context window. It’s designed for coding agents, game development, tool use, research, and other workflows where the model has to actually do the work, not just answer a prompt. And right now, you can try Hy4 preview for free on WorkBuddy until September 10. If you want to see how far an AI agent can take a real project, try it yourself: See what you can build with it. Tencent AI

I gave an AI one instruction: “Build me a playable 3D game from scratch.” Then I let it cook. 👀 I tested Tencent Hy4 preview inside WorkBuddy, and it went way beyond just generating a few snippets of code. It could: → Plan the game structure → Generate the code across files → Create the UI and interactions → Debug problems along the way → Iterate on the project while keeping context That last part is what caught my attention. Hy4 preview is built for these long-horizon tasks, with 770B total parameters, 49B active parameters, and a 1M-token context window. It’s designed for coding agents, game development, tool use, research, and other workflows where the model has to actually do the work, not just answer a prompt. And right now, you can try Hy4 preview for free on WorkBuddy until September 10. If you want to see how far an AI agent can take a real project, try it yourself: See what you can build with it. Tencent AI

Toha Khan

66,214 views • 1 month ago

Announcing the Open Model Series: a new series on open source AI models, with merve of Hugging Face. Hugging Face is the open model hub: it hosts model weights, datasets, evals, & demo apps, and maintains the open-source libraries (like transformers) used to run and train them. I think of it like GitHub, but for AI models. You can download open models and run them on your own machines. In this series we'll cover what open source and open weight models are, how they work, how to run them yourself, and why they're important. By the way - this video is edited entirely with Claude! I can make a tutorial on how I did it. If there's a question you want answered about open models, leave it in the comments. We will read them all. Episode 1 is out soon.

Announcing the Open Model Series: a new series on open source AI models, with merve of Hugging Face. Hugging Face is the open model hub: it hosts model weights, datasets, evals, & demo apps, and maintains the open-source libraries (like transformers) used to run and train them. I think of it like GitHub, but for AI models. You can download open models and run them on your own machines. In this series we'll cover what open source and open weight models are, how they work, how to run them yourself, and why they're important. By the way - this video is edited entirely with Claude! I can make a tutorial on how I did it. If there's a question you want answered about open models, leave it in the comments. We will read them all. Episode 1 is out soon.

Harper Carroll

30,392 views • 2 months ago

Vals AI CEO Rayan Krishnan on how they score a model's ability to build its own successor: "One benchmark we released recently is our Recursive Self-Improvement Index. It's a topic which a lot of the big labs have been talking about and starting to report on in their model cards." "There isn't a shared language to talk about the RSI potential of models. So we created this as an apples-to-apples way to actually benchmark across the models." "In an ideal world, what you want to do is take a frontier model and have it train the next version of itself and see where the delta comes from. But obviously that's very expensive and slow." "What we're doing is forming a set of proxies for every part of the process it takes to build the next version of the model." "There's work around pre-training, post-training, harness-level engineering, and then seeing in which mechanisms and behaviors the models are able to do very good research work and build something new, and where they're struggling." Rayan Krishnan in DC Jennifer Li

Vals AI CEO Rayan Krishnan on how they score a model's ability to build its own successor: "One benchmark we released recently is our Recursive Self-Improvement Index. It's a topic which a lot of the big labs have been talking about and starting to report on in their model cards." "There isn't a shared language to talk about the RSI potential of models. So we created this as an apples-to-apples way to actually benchmark across the models." "In an ideal world, what you want to do is take a frontier model and have it train the next version of itself and see where the delta comes from. But obviously that's very expensive and slow." "What we're doing is forming a set of proxies for every part of the process it takes to build the next version of the model." "There's work around pre-training, post-training, harness-level engineering, and then seeing in which mechanisms and behaviors the models are able to do very good research work and build something new, and where they're struggling." Rayan Krishnan in DC Jennifer Li

a16z

34,968 views • 1 month ago

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

133,587 views • 2 months ago

New short course on advanced retrieval for RAG (retrieval augmented generation)! RAG fetches relevant documents to give context to an LLM. In Advanced Retrieval for AI with Chroma, taught by Chroma founder anton 🇺🇸, you’ll learn: (i) Query expansion using an LLM to rewrite and improve a query, by either generating either additional relevant queries or a hypothetical answer to the query. (ii) Reranking using a cross-encoder - a model trained to measure similarity between two inputs presented simultaneously. Reranking reorders retrieved documents based on the cross-encoder similarity measure. (iii) Constructing and training an Embedding Adaptor, which is a model that adapts the embedding values to be more relevant to your use case. Each of these techniques can help you build much better RAG systems. Please sign up for the course here:

New short course on advanced retrieval for RAG (retrieval augmented generation)! RAG fetches relevant documents to give context to an LLM. In Advanced Retrieval for AI with Chroma, taught by Chroma founder anton 🇺🇸, you’ll learn: (i) Query expansion using an LLM to rewrite and improve a query, by either generating either additional relevant queries or a hypothetical answer to the query. (ii) Reranking using a cross-encoder - a model trained to measure similarity between two inputs presented simultaneously. Reranking reorders retrieved documents based on the cross-encoder similarity measure. (iii) Constructing and training an Embedding Adaptor, which is a model that adapts the embedding values to be more relevant to your use case. Each of these techniques can help you build much better RAG systems. Please sign up for the course here:

Andrew Ng

191,219 views • 2 years ago

You can build your 3D models directly from a prompt into Claude code. All thanks to this beautiful python project, build123 a parametric modeling framework for 2D and 3D CAD. It is built on the Open Cascade geometric kernel and it is suitable for 3D printing.

You can build your 3D models directly from a prompt into Claude code. All thanks to this beautiful python project, build123 a parametric modeling framework for 2D and 3D CAD. It is built on the Open Cascade geometric kernel and it is suitable for 3D printing.

Marco Franzon

20,915 views • 6 months ago

I recorded a video on how you can build a complete RAG application without writing any code. I used Langflow, an open-source project that sits on top of Langchain. Langflow hides all the complexity and makes the process 10x easier. This is a step-by-step tutorial.

I recorded a video on how you can build a complete RAG application without writing any code. I used Langflow, an open-source project that sits on top of Langchain. Langflow hides all the complexity and makes the process 10x easier. This is a step-by-step tutorial.

Santiago

96,267 views • 2 years ago

This AI agent does the job of an F1 TV director, and you can see every choice it makes. It follows qualifying at the 2026 Dutch GP, with live timing and onboard cameras from every car, and picks which shots to put on screen. Here's what it runs on: a LangChain deep agent that plans each step and uses tools NVIDIA Nemotron as the model, running on Nebius Token Factory through an OpenAI-compatible API LangSmith, which records every run The LangSmith part is my favorite. You can open any run and see which model call it made, which tool it used, and how long each step took. If something goes wrong, you can find the exact step instead of guessing. Each piece comes from a different company, and you can switch any one of them for another tool without rebuilding the rest. This build is part of the Nebius AI Builder Program launch, and you can use the same pieces to build your own. Join the Nebius AI Builder Program free at $400+ in credits across the stack on day one. In collaboration with Nebius #nebiuspartnership

This AI agent does the job of an F1 TV director, and you can see every choice it makes. It follows qualifying at the 2026 Dutch GP, with live timing and onboard cameras from every car, and picks which shots to put on screen. Here's what it runs on: a LangChain deep agent that plans each step and uses tools NVIDIA Nemotron as the model, running on Nebius Token Factory through an OpenAI-compatible API LangSmith, which records every run The LangSmith part is my favorite. You can open any run and see which model call it made, which tool it used, and how long each step took. If something goes wrong, you can find the exact step instead of guessing. Each piece comes from a different company, and you can switch any one of them for another tool without rebuilding the rest. This build is part of the Nebius AI Builder Program launch, and you can use the same pieces to build your own. Join the Nebius AI Builder Program free at $400+ in credits across the stack on day one. In collaboration with Nebius #nebiuspartnership

Chidanand Tripathi

99,060 views • 20 days ago

1/ First, how do you benchmark an AI model? You give different models the same test cases, and score them. You can do the same with voice models using Word Error Rate (WER). You generate the audio, transcribe it with a speech recognition (ASR) model, and compare that transcript to the original text. And this tells you whether the model said the right words. But the same sentence can be said correctly, while still being wrong in context.

1/ First, how do you benchmark an AI model? You give different models the same test cases, and score them. You can do the same with voice models using Word Error Rate (WER). You generate the audio, transcribe it with a speech recognition (ASR) model, and compare that transcript to the original text. And this tells you whether the model said the right words. But the same sentence can be said correctly, while still being wrong in context.

Cartesia

10,184 views • 29 days ago

99% of AI applications are cool-looking demos. Impressive, but don't get fooled by the hype. It takes a lot to build enterprise-grade products that deliver real value. I have at least three weekly conversations with companies that want to use a Large Language Model with their data. The demand is huge! Here is one idea about what you can do to help. The use cases that most of these companies want to solve are similar: They have an extensive knowledge base and want to build a simple application that uses that information to answer questions. In other words, they need help building Retrieval Augmented Generation (RAG) applications they can use in many different scenarios: 1. To train new employees 2. To help their support team 3. To search old meetings and documents 4. To help with their research However, building these systems is not straightforward. Yes, there's a lot of information online, but there aren't enough people who know how to create solutions that work. Here is the idea: Today, you can build an enterprise-grade RAG application without writing code. A couple of MIT PhDs with 10+ years of experience building AI applications created . It's a no-code platform for building applications using Large Language Models. They are partnering with me on this post. You can use Stack AI to create, test, and deploy an end-to-end production-ready AI system. It's SOC-2, HIPAA, and GDPR compliant and offers SSO, role management, access control, and on-premise deployments. Of course, you can use the platform with any LLM on the market now. It's the whole nine yards for building AI applications. Check them out here: 2023 was about models. 2024 is about the tools using these models to build production-ready applications. That's where I'd start.

99% of AI applications are cool-looking demos. Impressive, but don't get fooled by the hype. It takes a lot to build enterprise-grade products that deliver real value. I have at least three weekly conversations with companies that want to use a Large Language Model with their data. The demand is huge! Here is one idea about what you can do to help. The use cases that most of these companies want to solve are similar: They have an extensive knowledge base and want to build a simple application that uses that information to answer questions. In other words, they need help building Retrieval Augmented Generation (RAG) applications they can use in many different scenarios: 1. To train new employees 2. To help their support team 3. To search old meetings and documents 4. To help with their research However, building these systems is not straightforward. Yes, there's a lot of information online, but there aren't enough people who know how to create solutions that work. Here is the idea: Today, you can build an enterprise-grade RAG application without writing code. A couple of MIT PhDs with 10+ years of experience building AI applications created . It's a no-code platform for building applications using Large Language Models. They are partnering with me on this post. You can use Stack AI to create, test, and deploy an end-to-end production-ready AI system. It's SOC-2, HIPAA, and GDPR compliant and offers SSO, role management, access control, and on-premise deployments. Of course, you can use the platform with any LLM on the market now. It's the whole nine yards for building AI applications. Check them out here: 2023 was about models. 2024 is about the tools using these models to build production-ready applications. That's where I'd start.

Santiago

197,702 views • 2 years ago

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Learn how to build an optimized LLM inference system from the ground up in our new short course, Efficiently Serving LLMs, built in collaboration with Predibase by Rubrik and taught by Travis Addair. Whether you're serving your own LLM or using a model hosting service, this course will give you a deep understanding of the optimizations required to efficiently serve many users at once. - Learn how LLMs generate text one token at a time, and how techniques like KV caching, continuous batching, and quantization speed things up and optimize memory usage for serving multiple users. - Benchmark the performance of these LLM optimizations to explore the trade-offs between quickly responding to an individual user’s request vs. serving many users at once. - Use techniques like low-rank adaptation (LoRA) to efficiently serve hundreds of unique, custom fine-tuned models on a single device, without sacrificing throughput. - Use Predibase's LoRAX framework to see optimization techniques in action on a real LLM server. Sign up here:

Andrew Ng

104,727 views • 2 years ago