Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introducing Vals-Smith: turn your code base into a customized benchmark. Public benchmarks tell you which model is strongest overall, not which model is the best on your code. Vals-Smith turns your merged pull requests into real coding tasks and measures the percentage a model can actually resolve. New models... show more

Vals AI

28,754 subscribers

46,847 views • 3 months ago •via X (Twitter)

Science & Technology Education Health & Wellness

Anya Rossi• Live Now

Private livecam show

28 Comments

Vals AI3 months ago

Join the waitlist today!

Rayan Krishnan3 months ago

props to @OfirPress @jyangballin @KLieret for laying a lot of the ground work. by default we evaluate all models with mini swe agent, methodology is heavily swe bench inspired

Jerry Liu3 months ago

congrats on the launch!!

Gabriel Stengel3 months ago

Congrats @RayanKrishnan on the launch!

Rez Havaei3 months ago

Great work!

Aryo Patel3 months ago

Awesome release -- can't wait to try it out code speed about to double

Jihoon3 months ago

i had this idea for months and these days its great, you just have to wait because good ideas end up getting implemented thanks!

Braeden Norris3 months ago

Super cool launch, congrats!

Inflectiv AI ⧉3 months ago

🔥🔥

安叫兽|Bird🕊️ 🔶 BNB3 months ago

拿自己仓库测，比看榜单踏实多了

Ronak Malde3 months ago

This is very exciting!!

icosaedro3 months ago

PRs as the benchmark set is the only metric that matters. Generic evals lie.

Sumer Sao3 months ago

congrats guys!

Jason Zhao3 months ago

Awesome stuff

Rona Wang3 months ago

🚀

Vivek Gopalan3 months ago

Really important

Moritz Stephan3 months ago

this is awesome

Daniel Rabinovich3 months ago

This is really cool!

Faraz3 months ago

Really cool work @RayanKrishnan !!

Dhruv Mangtani3 months ago

sounds expensive lol

Sergey Karayev3 months ago

repo-specific evals are the way. we built this into @Superconductor too, and you can run your own on merged PRs today no waitlist:

Dhruvik Parikh 🌎3 months ago

🔥🔥🔥

shyamal3 months ago

congrats on the launch!

iTokens.io ⚡️ AI API3 months ago

This feels like the direction coding evaluation is heading. The most useful benchmark isn't the one that ranks models globally—it's the one built from your own workflows and codebase.

Jihoon3 months ago

@grok competitor landscape in both closed and open source

RAZA | AI EXPLORER3 months ago

Measuring performance on your own code is a game-changer. Love this.

Lucas Campa3 months ago

Looks awesome 🚀🚀

Panizzutti3 months ago

I like this but vibe coded video

Related Videos

Introducing Arena Mode in Windsurf: One prompt. Two models. Your vote. Benchmarks don't reflect real-world coding quality. The best model for you depends on your codebase and stack. So we made real-world coding the benchmark. Free for the next week. May the best model win.

Introducing Arena Mode in Windsurf: One prompt. Two models. Your vote. Benchmarks don't reflect real-world coding quality. The best model for you depends on your codebase and stack. So we made real-world coding the benchmark. Free for the next week. May the best model win.

Devin Developers

1,073,143 views • 8 months ago

🧳 Bring Your Own AI Models to GitHub Copilot in VS Code Use your own Azure-hosted AI models with GitHub Copilot in VS Code. See how Bring Your Own Key (BYOK) lets you connect models from Microsoft Foundry, choose which models appear in Copilot, and use the right model for different coding tasks.

🧳 Bring Your Own AI Models to GitHub Copilot in VS Code Use your own Azure-hosted AI models with GitHub Copilot in VS Code. See how Bring Your Own Key (BYOK) lets you connect models from Microsoft Foundry, choose which models appear in Copilot, and use the right model for different coding tasks.

Visual Studio Code

38,536 views • 1 month ago

They're still your ideas. [code]smith just writes the code. [code]smith is a cloud software engineer that works alongside you. Autofixing PRs, CI failures and reviews.

They're still your ideas. [code]smith just writes the code. [code]smith is a cloud software engineer that works alongside you. Autofixing PRs, CI failures and reviews.

Blacksmith

3,621,539 views • 2 months ago

SOMEONE MADE IT SO YOU CAN CODE WITH 174 AI MODELS FROM 23 PROVIDERS FOR FREE one npm package and one install: "npm i -g free-coding-models" switch between any model instantly. benchmark them against each other in real time to see which one actually codes best for your use case. no subscriptions. no API keys. no cost. 174 models. 23 providers, all free, all from your terminal. if you're tired of paying for 3 different AI subscriptions just to compare which model codes better, this is it

SOMEONE MADE IT SO YOU CAN CODE WITH 174 AI MODELS FROM 23 PROVIDERS FOR FREE one npm package and one install: "npm i -g free-coding-models" switch between any model instantly. benchmark them against each other in real time to see which one actually codes best for your use case. no subscriptions. no API keys. no cost. 174 models. 23 providers, all free, all from your terminal. if you're tired of paying for 3 different AI subscriptions just to compare which model codes better, this is it

Om Patel

93,660 views • 6 months ago

Introducing Shiro - the CUTEST ai coding agent which turns your prompts into pull requests. Link in the comments. @JoinShiro

Introducing Shiro - the CUTEST ai coding agent which turns your prompts into pull requests. Link in the comments. @JoinShiro

Dev

496,344 views • 1 year ago

This approach has made Sonnet the model of choice for developers worldwide. In addition to our new model, we're launching Claude Code, our first coding tool, in a limited research preview. With Claude Code, you can delegate substantial tasks to Claude—right from your terminal.

This approach has made Sonnet the model of choice for developers worldwide. In addition to our new model, we're launching Claude Code, our first coding tool, in a limited research preview. With Claude Code, you can delegate substantial tasks to Claude—right from your terminal.

Anthropic

1,141,101 views • 1 year ago

Introducing Claude Code on the web. You can now delegate coding tasks to Claude without opening your terminal.

Introducing Claude Code on the web. You can now delegate coding tasks to Claude without opening your terminal.

Claude

2,666,443 views • 1 year ago

Meet AutoRouter, the most intelligent way to pick models You’re wasting tokens selecting frontier models for every prompt, and sometimes for worse performance AutoRouter handles the decision layer: - Understands the prompt request - Picks the best model fit for the task - Saves you money at every turn - Plugs into your current setup Works with your plans & every leading model inside Codex/Claude Code Beta is live:

Meet AutoRouter, the most intelligent way to pick models You’re wasting tokens selecting frontier models for every prompt, and sometimes for worse performance AutoRouter handles the decision layer: - Understands the prompt request - Picks the best model fit for the task - Saves you money at every turn - Plugs into your current setup Works with your plans & every leading model inside Codex/Claude Code Beta is live:

ty

82,364 views • 2 days ago

🎙️ Dictate in Multiple Languages in #VSCode Dictation can detect your language and turn your speech into text using a local model. You can use it in Chat, the editor, and the integrated terminal.

🎙️ Dictate in Multiple Languages in #VSCode Dictation can detect your language and turn your speech into text using a local model. You can use it in Chat, the editor, and the integrated terminal.

Visual Studio Code

27,733 views • 2 months ago

Multimodal AI is here 🤯 GPT-4 can now turn your images into a text file in a snap with the new code interpreter model. Witness the OCR magic in action 🔥

Multimodal AI is here 🤯 GPT-4 can now turn your images into a text file in a snap with the new code interpreter model. Witness the OCR magic in action 🔥

Shubham Saboo

727,673 views • 3 years ago

Introducing OpenResearch: run parallel research agents with any model Bring your own compute and your code + data stays with you A virtual lab on your machine

Introducing OpenResearch: run parallel research agents with any model Bring your own compute and your code + data stays with you A virtual lab on your machine

alphaXiv

32,864 views • 3 months ago

Don't talk directly to a model. You can make your application 10x more flexible by simply adding one intermediate layer between your code and the model. This is the simplest trick you can use to improve the architecture of whatever you are building.

Don't talk directly to a model. You can make your application 10x more flexible by simply adding one intermediate layer between your code and the model. This is the simplest trick you can use to improve the architecture of whatever you are building.

Santiago

88,884 views • 6 months ago

Introducing Merge Gateway - Build Your Own Router. You're three sprints into your coding assistant. You pick the most hyped model, integrate, test, deploy. A month later, a new model drops. Now you re-test, re-integrate, re-deploy. Your product didn't change, but the benchmark did. That's how most AI teams operate. Chasing a "best" defined by people who've never seen their product. There is no best model. There's only the right one for your product, users, and use-cases. Build Your Own Router runs on your definition of good. Pick your benchmarks, weigh them, add your own evals. Merge routes every request to your winner. 👉$100 in credits to the first 200 people that comment

Introducing Merge Gateway - Build Your Own Router. You're three sprints into your coding assistant. You pick the most hyped model, integrate, test, deploy. A month later, a new model drops. Now you re-test, re-integrate, re-deploy. Your product didn't change, but the benchmark did. That's how most AI teams operate. Chasing a "best" defined by people who've never seen their product. There is no best model. There's only the right one for your product, users, and use-cases. Build Your Own Router runs on your definition of good. Pick your benchmarks, weigh them, add your own evals. Merge routes every request to your winner. 👉$100 in credits to the first 200 people that comment

Shensi Ding

3,668,527 views • 5 months ago

Introducing WrapdCode Claude Code with Bring Your Own Model 1. Use GPT, Gemini, Llama, Mistral - 300+ models 2. Same Claude Code UI, zero changes 3. Plug in your API key and go Stop being stuck on one model.

Introducing WrapdCode Claude Code with Bring Your Own Model 1. Use GPT, Gemini, Llama, Mistral - 300+ models 2. Same Claude Code UI, zero changes 3. Plug in your API key and go Stop being stuck on one model.

aditya

29,014 views • 6 months ago

Fusion Models by robinhood:0xa92768863a55d8a0591709f7f5e594a249d36ea3, explained properly. Why a panel beats one model: Ask three models the same hard question and they disagree. The disagreement is the information, where they contradict each other is where one of them is wrong, and a judge can see it. OpenRouter shipped exactly this in June and their panels beat any single frontier model. Credit to them and their $10B sale to stripe. Why the same panel for everyone is the wrong shape: You don't have one best model. You have a best model per job. The one you trust for code is not the one you trust for writing, and neither is the one you'd pick for a five-line reply. A panel chosen per request, identical for every user, can't know that. A panel arranged around you can. The number that matters isn't price: It's cost per answer that works. Model A costs $4 per million tokens and gets your job right one time in four. Model B costs $10 and gets it right three times in four. Per answer you actually keep, A costs $16 and B costs $13. The cheap model was the expensive one. Everyone comparing price lists is measuring the wrong thing. You never rate anything: Stopping an answer halfway is a rating. Regenerating is a rating. Switching model and asking again is the loudest rating there is. Every one of those already writes a row we can learn from. The router learns which models earn your credits, per job, and tries a cheaper one now and then to check. Fusion only when it pays: Cost mode tries the cheap model first and escalates only if a judge says it failed. Performance mode fans out to your panel and merges. Most simple prompts never fuse. That's why it costs less, not more. Put it together and you have a personal model: Not trained. Assembled. Your profile from a two-minute questionnaire and everything you've imported from Claude or ChatGPT. Your memory, shared across every model so switching never starts from zero. Your skills, workflows you build once and run on any model. Your scorecard, which models earn your credits per job. One per person. Nobody else has yours. Every model is someone else's; the arrangement is you, and it leaves with you. People aren't loyal to models. They're loyal to results. A moat can't be a commodity, but it can be our own creation.

askr

13,900 views • 1 month ago

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

133,793 views • 2 months ago

Introducing Cursor 2.0. Our first coding model and the best way to code with agents.

Introducing Cursor 2.0. Our first coding model and the best way to code with agents.

Cursor

3,572,368 views • 1 year ago

Stop hardcoding one model name in your code. Now you can give each request a policy: a small rule for what the call needs It picks the right model for the job, on your own keys This is unhardcoded, our new open source routing for AI models, live today

Stop hardcoding one model name in your code. Now you can give each request a policy: a small rule for what the call needs It picks the right model for the job, on your own keys This is unhardcoded, our new open source routing for AI models, live today

GenLayer

98,139 views • 4 months ago

Run your coding agent on a local model, with QVAC. OpenCode now plugs into a model on your own machine through our official plugin. That changes two things. Privacy by default. Your repo, your prompts, and every command the agent runs stay on your device. No third-party model ever sees your code. For an agent that reads your codebase and runs commands, that is the point, not a nice-to-have. New workflows. With no meter and no rate limit, the agent does not have to stop. Let it loop on a task, refactor a whole repo overnight, or run a watch-build-fix loop that keeps going. Setup takes a few minutes: one line in opencode.json and OpenCode runs on a local QVAC model. In the video demo, it handles three real developer tasks back to back, fully offline.

Run your coding agent on a local model, with QVAC. OpenCode now plugs into a model on your own machine through our official plugin. That changes two things. Privacy by default. Your repo, your prompts, and every command the agent runs stay on your device. No third-party model ever sees your code. For an agent that reads your codebase and runs commands, that is the point, not a nice-to-have. New workflows. With no meter and no rate limit, the agent does not have to stop. Let it loop on a task, refactor a whole repo overnight, or run a watch-build-fix loop that keeps going. Setup takes a few minutes: one line in opencode.json and OpenCode runs on a local QVAC model. In the video demo, it handles three real developer tasks back to back, fully offline.

QVAC

11,382,936 views • 4 months ago

STOP GUESSING WHICH LOCAL AI MODELS YOUR COMPUTER CAN RUN. LLMFIT SCANS YOUR HARDWARE AND TELLS YOU WHICH MODELS WILL ACTUALLY RUN WELL, INCLUDING THE BEST QUANTIZATION TO USE.

STOP GUESSING WHICH LOCAL AI MODELS YOUR COMPUTER CAN RUN. LLMFIT SCANS YOUR HARDWARE AND TELLS YOU WHICH MODELS WILL ACTUALLY RUN WELL, INCLUDING THE BEST QUANTIZATION TO USE.

0xMarioNawfal

48,706 views • 1 month ago