Загрузка видео...

Не удалось загрузить видео

На главную

Introducing Vals-Smith: turn your code base into a customized benchmark. Public benchmarks tell you which model is strongest overall, not which model is the best on your code. Vals-Smith turns your merged pull requests into real coding tasks and measures the percentage a model can actually resolve. New models...

ValsAI's profile picture

Vals AI

28,754 subscribers

46,847 просмотров • 3 месяцев назад •via X (Twitter)

Наука и технологииОбразованиеЗдоровье и велнес
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 28

Фото профиля Vals AI
Vals AI3 месяцев назад

Join the waitlist today!

Фото профиля Rayan Krishnan
Rayan Krishnan3 месяцев назад

props to @OfirPress @jyangballin @KLieret for laying a lot of the ground work. by default we evaluate all models with mini swe agent, methodology is heavily swe bench inspired

Фото профиля Jerry Liu
Jerry Liu3 месяцев назад

congrats on the launch!!

Фото профиля Gabriel Stengel
Gabriel Stengel3 месяцев назад

Congrats @RayanKrishnan on the launch!

Фото профиля Rez Havaei
Rez Havaei3 месяцев назад

Great work!

Фото профиля Aryo Patel
Aryo Patel3 месяцев назад

Awesome release -- can't wait to try it out code speed about to double

Фото профиля Jihoon
Jihoon3 месяцев назад

i had this idea for months and these days its great, you just have to wait because good ideas end up getting implemented thanks!

Фото профиля Braeden Norris
Braeden Norris3 месяцев назад

Super cool launch, congrats!

Фото профиля Inflectiv AI ⧉
Inflectiv AI ⧉3 месяцев назад

🔥🔥

Фото профиля 安叫兽|Bird🕊️ 🔶 BNB
安叫兽|Bird🕊️ 🔶 BNB3 месяцев назад

拿自己仓库测，比看榜单踏实多了

Фото профиля Ronak Malde
Ronak Malde3 месяцев назад

This is very exciting!!

Фото профиля icosaedro
icosaedro3 месяцев назад

PRs as the benchmark set is the only metric that matters. Generic evals lie.

Фото профиля Sumer Sao
Sumer Sao3 месяцев назад

congrats guys!

Фото профиля Jason Zhao
Jason Zhao3 месяцев назад

Awesome stuff

Фото профиля Rona Wang
Rona Wang3 месяцев назад

🚀

Фото профиля Vivek Gopalan
Vivek Gopalan3 месяцев назад

Really important

Фото профиля Moritz Stephan
Moritz Stephan3 месяцев назад

this is awesome

Фото профиля Daniel Rabinovich
Daniel Rabinovich3 месяцев назад

This is really cool!

Фото профиля Faraz
Faraz3 месяцев назад

Really cool work @RayanKrishnan !!

Фото профиля Dhruv Mangtani
Dhruv Mangtani3 месяцев назад

sounds expensive lol

Фото профиля Sergey Karayev
Sergey Karayev3 месяцев назад

repo-specific evals are the way. we built this into @Superconductor too, and you can run your own on merged PRs today no waitlist:

Фото профиля Dhruvik Parikh 🌎
Dhruvik Parikh 🌎3 месяцев назад

🔥🔥🔥

Фото профиля shyamal
shyamal3 месяцев назад

congrats on the launch!

Фото профиля iTokens.io ⚡️ AI API
iTokens.io ⚡️ AI API3 месяцев назад

This feels like the direction coding evaluation is heading. The most useful benchmark isn't the one that ranks models globally—it's the one built from your own workflows and codebase.

Фото профиля Jihoon
Jihoon3 месяцев назад

@grok competitor landscape in both closed and open source

Фото профиля RAZA | AI EXPLORER
RAZA | AI EXPLORER3 месяцев назад

Measuring performance on your own code is a game-changer. Love this.

Фото профиля Lucas Campa
Lucas Campa3 месяцев назад

Looks awesome 🚀🚀

Фото профиля Panizzutti
Panizzutti3 месяцев назад

I like this but vibe coded video

Похожие видео

Introducing Arena Mode in Windsurf: One prompt. Two models. Your vote. Benchmarks don't reflect real-world coding quality. The best model for you depends on your codebase and stack. So we made real-world coding the benchmark. Free for the next week. May the best model win.
0:33

Introducing Arena Mode in Windsurf: One prompt. Two models. Your vote. Benchmarks don't reflect real-world coding quality. The best model for you depends on your codebase and stack. So we made real-world coding the benchmark. Free for the next week. May the best model win.

Devin Developers

1,073,143 просмотров • 8 месяцев назад

🧳 Bring Your Own AI Models to GitHub Copilot in VS Code Use your own Azure-hosted AI models with GitHub Copilot in VS Code. See how Bring Your Own Key (BYOK) lets you connect models from Microsoft Foundry, choose which models appear in Copilot, and use the right model for different coding tasks.
1:39

🧳 Bring Your Own AI Models to GitHub Copilot in VS Code Use your own Azure-hosted AI models with GitHub Copilot in VS Code. See how Bring Your Own Key (BYOK) lets you connect models from Microsoft Foundry, choose which models appear in Copilot, and use the right model for different coding tasks.

Visual Studio Code

38,536 просмотров • 1 месяц назад

They're still your ideas. [code]smith just writes the code. [code]smith is a cloud software engineer that works alongside you. Autofixing PRs, CI failures and reviews.
0:34

They're still your ideas. [code]smith just writes the code. [code]smith is a cloud software engineer that works alongside you. Autofixing PRs, CI failures and reviews.

Blacksmith

3,621,539 просмотров • 2 месяцев назад

SOMEONE MADE IT SO YOU CAN CODE WITH 174 AI MODELS FROM 23 PROVIDERS FOR FREE one npm package and one install: "npm i -g free-coding-models" switch between any model instantly. benchmark them against each other in real time to see which one actually codes best for your use case. no subscriptions. no API keys. no cost. 174 models. 23 providers, all free, all from your terminal. if you're tired of paying for 3 different AI subscriptions just to compare which model codes better, this is it
2:55

SOMEONE MADE IT SO YOU CAN CODE WITH 174 AI MODELS FROM 23 PROVIDERS FOR FREE one npm package and one install: "npm i -g free-coding-models" switch between any model instantly. benchmark them against each other in real time to see which one actually codes best for your use case. no subscriptions. no API keys. no cost. 174 models. 23 providers, all free, all from your terminal. if you're tired of paying for 3 different AI subscriptions just to compare which model codes better, this is it

Om Patel

93,660 просмотров • 6 месяцев назад

Introducing Shiro - the CUTEST ai coding agent which turns your prompts into pull requests. Link in the comments. @JoinShiro
1:58

Introducing Shiro - the CUTEST ai coding agent which turns your prompts into pull requests. Link in the comments. @JoinShiro

Dev

496,344 просмотров • 1 год назад

This approach has made Sonnet the model of choice for developers worldwide. In addition to our new model, we're launching Claude Code, our first coding tool, in a limited research preview. With Claude Code, you can delegate substantial tasks to Claude—right from your terminal.
3:54

This approach has made Sonnet the model of choice for developers worldwide. In addition to our new model, we're launching Claude Code, our first coding tool, in a limited research preview. With Claude Code, you can delegate substantial tasks to Claude—right from your terminal.

Anthropic

1,141,101 просмотров • 1 год назад

Introducing Claude Code on the web. You can now delegate coding tasks to Claude without opening your terminal.
1:00

Introducing Claude Code on the web. You can now delegate coding tasks to Claude without opening your terminal.

Claude

2,666,443 просмотров • 1 год назад

Meet AutoRouter, the most intelligent way to pick models You’re wasting tokens selecting frontier models for every prompt, and sometimes for worse performance AutoRouter handles the decision layer: - Understands the prompt request - Picks the best model fit for the task - Saves you money at every turn - Plugs into your current setup Works with your plans & every leading model inside Codex/Claude Code Beta is live:
0:44

Meet AutoRouter, the most intelligent way to pick models You’re wasting tokens selecting frontier models for every prompt, and sometimes for worse performance AutoRouter handles the decision layer: - Understands the prompt request - Picks the best model fit for the task - Saves you money at every turn - Plugs into your current setup Works with your plans & every leading model inside Codex/Claude Code Beta is live:

ty

82,364 просмотров • 2 дней назад

🎙️ Dictate in Multiple Languages in #VSCode Dictation can detect your language and turn your speech into text using a local model. You can use it in Chat, the editor, and the integrated terminal.
0:41

🎙️ Dictate in Multiple Languages in #VSCode Dictation can detect your language and turn your speech into text using a local model. You can use it in Chat, the editor, and the integrated terminal.

Visual Studio Code

27,733 просмотров • 2 месяцев назад

Multimodal AI is here 🤯 GPT-4 can now turn your images into a text file in a snap with the new code interpreter model. Witness the OCR magic in action 🔥
0:29

Multimodal AI is here 🤯 GPT-4 can now turn your images into a text file in a snap with the new code interpreter model. Witness the OCR magic in action 🔥

Shubham Saboo

727,673 просмотров • 3 лет назад

Introducing OpenResearch: run parallel research agents with any model Bring your own compute and your code + data stays with you A virtual lab on your machine
0:56

Introducing OpenResearch: run parallel research agents with any model Bring your own compute and your code + data stays with you A virtual lab on your machine

alphaXiv

32,864 просмотров • 3 месяцев назад

Don't talk directly to a model. You can make your application 10x more flexible by simply adding one intermediate layer between your code and the model. This is the simplest trick you can use to improve the architecture of whatever you are building.
9:47

Don't talk directly to a model. You can make your application 10x more flexible by simply adding one intermediate layer between your code and the model. This is the simplest trick you can use to improve the architecture of whatever you are building.

Santiago

88,884 просмотров • 6 месяцев назад

Introducing Merge Gateway - Build Your Own Router. You're three sprints into your coding assistant. You pick the most hyped model, integrate, test, deploy. A month later, a new model drops. Now you re-test, re-integrate, re-deploy. Your product didn't change, but the benchmark did. That's how most AI teams operate. Chasing a "best" defined by people who've never seen their product. There is no best model. There's only the right one for your product, users, and use-cases. Build Your Own Router runs on your definition of good. Pick your benchmarks, weigh them, add your own evals. Merge routes every request to your winner. 👉$100 in credits to the first 200 people that comment
1:57

Introducing Merge Gateway - Build Your Own Router. You're three sprints into your coding assistant. You pick the most hyped model, integrate, test, deploy. A month later, a new model drops. Now you re-test, re-integrate, re-deploy. Your product didn't change, but the benchmark did. That's how most AI teams operate. Chasing a "best" defined by people who've never seen their product. There is no best model. There's only the right one for your product, users, and use-cases. Build Your Own Router runs on your definition of good. Pick your benchmarks, weigh them, add your own evals. Merge routes every request to your winner. 👉$100 in credits to the first 200 people that comment

Shensi Ding

3,668,527 просмотров • 5 месяцев назад

Introducing WrapdCode Claude Code with Bring Your Own Model 1. Use GPT, Gemini, Llama, Mistral - 300+ models 2. Same Claude Code UI, zero changes 3. Plug in your API key and go Stop being stuck on one model.
0:26

Introducing WrapdCode Claude Code with Bring Your Own Model 1. Use GPT, Gemini, Llama, Mistral - 300+ models 2. Same Claude Code UI, zero changes 3. Plug in your API key and go Stop being stuck on one model.

aditya

29,014 просмотров • 6 месяцев назад

Fusion Models by robinhood:0xa92768863a55d8a0591709f7f5e594a249d36ea3, explained properly. Why a panel beats one model: Ask three models the same hard question and they disagree. The disagreement is the information, where they contradict each other is where one of them is wrong, and a judge can see it. OpenRouter shipped exactly this in June and their panels beat any single frontier model. Credit to them and their $10B sale to stripe. Why the same panel for everyone is the wrong shape: You don't have one best model. You have a best model per job. The one you trust for code is not the one you trust for writing, and neither is the one you'd pick for a five-line reply. A panel chosen per request, identical for every user, can't know that. A panel arranged around you can. The number that matters isn't price: It's cost per answer that works. Model A costs $4 per million tokens and gets your job right one time in four. Model B costs $10 and gets it right three times in four. Per answer you actually keep, A costs $16 and B costs $13. The cheap model was the expensive one. Everyone comparing price lists is measuring the wrong thing. You never rate anything: Stopping an answer halfway is a rating. Regenerating is a rating. Switching model and asking again is the loudest rating there is. Every one of those already writes a row we can learn from. The router learns which models earn your credits, per job, and tries a cheaper one now and then to check. Fusion only when it pays: Cost mode tries the cheap model first and escalates only if a judge says it failed. Performance mode fans out to your panel and merges. Most simple prompts never fuse. That's why it costs less, not more. Put it together and you have a personal model: Not trained. Assembled. Your profile from a two-minute questionnaire and everything you've imported from Claude or ChatGPT. Your memory, shared across every model so switching never starts from zero. Your skills, workflows you build once and run on any model. Your scorecard, which models earn your credits per job. One per person. Nobody else has yours. Every model is someone else's; the arrangement is you, and it leaves with you. People aren't loyal to models. They're loyal to results. A moat can't be a commodity, but it can be our own creation.
0:46

Fusion Models by robinhood:0xa92768863a55d8a0591709f7f5e594a249d36ea3, explained properly. Why a panel beats one model: Ask three models the same hard question and they disagree. The disagreement is the information, where they contradict each other is where one of them is wrong, and a judge can see it. OpenRouter shipped exactly this in June and their panels beat any single frontier model. Credit to them and their $10B sale to stripe. Why the same panel for everyone is the wrong shape: You don't have one best model. You have a best model per job. The one you trust for code is not the one you trust for writing, and neither is the one you'd pick for a five-line reply. A panel chosen per request, identical for every user, can't know that. A panel arranged around you can. The number that matters isn't price: It's cost per answer that works. Model A costs $4 per million tokens and gets your job right one time in four. Model B costs $10 and gets it right three times in four. Per answer you actually keep, A costs $16 and B costs $13. The cheap model was the expensive one. Everyone comparing price lists is measuring the wrong thing. You never rate anything: Stopping an answer halfway is a rating. Regenerating is a rating. Switching model and asking again is the loudest rating there is. Every one of those already writes a row we can learn from. The router learns which models earn your credits, per job, and tries a cheaper one now and then to check. Fusion only when it pays: Cost mode tries the cheap model first and escalates only if a judge says it failed. Performance mode fans out to your panel and merges. Most simple prompts never fuse. That's why it costs less, not more. Put it together and you have a personal model: Not trained. Assembled. Your profile from a two-minute questionnaire and everything you've imported from Claude or ChatGPT. Your memory, shared across every model so switching never starts from zero. Your skills, workflows you build once and run on any model. Your scorecard, which models earn your credits per job. One per person. Nobody else has yours. Every model is someone else's; the arrangement is you, and it leaves with you. People aren't loyal to models. They're loyal to results. A moat can't be a commodity, but it can be our own creation.

askr

13,900 просмотров • 1 месяц назад

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at
2:33

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

133,793 просмотров • 2 месяцев назад

Introducing Cursor 2.0. Our first coding model and the best way to code with agents.
1:04

Introducing Cursor 2.0. Our first coding model and the best way to code with agents.

Cursor

3,572,368 просмотров • 1 год назад

Stop hardcoding one model name in your code. Now you can give each request a policy: a small rule for what the call needs It picks the right model for the job, on your own keys This is unhardcoded, our new open source routing for AI models, live today
0:47

Stop hardcoding one model name in your code. Now you can give each request a policy: a small rule for what the call needs It picks the right model for the job, on your own keys This is unhardcoded, our new open source routing for AI models, live today

GenLayer

98,139 просмотров • 4 месяцев назад

Run your coding agent on a local model, with QVAC. OpenCode now plugs into a model on your own machine through our official plugin. That changes two things. Privacy by default. Your repo, your prompts, and every command the agent runs stay on your device. No third-party model ever sees your code. For an agent that reads your codebase and runs commands, that is the point, not a nice-to-have. New workflows. With no meter and no rate limit, the agent does not have to stop. Let it loop on a task, refactor a whole repo overnight, or run a watch-build-fix loop that keeps going. Setup takes a few minutes: one line in opencode.json and OpenCode runs on a local QVAC model. In the video demo, it handles three real developer tasks back to back, fully offline.
1:53

Run your coding agent on a local model, with QVAC. OpenCode now plugs into a model on your own machine through our official plugin. That changes two things. Privacy by default. Your repo, your prompts, and every command the agent runs stay on your device. No third-party model ever sees your code. For an agent that reads your codebase and runs commands, that is the point, not a nice-to-have. New workflows. With no meter and no rate limit, the agent does not have to stop. Let it loop on a task, refactor a whole repo overnight, or run a watch-build-fix loop that keeps going. Setup takes a few minutes: one line in opencode.json and OpenCode runs on a local QVAC model. In the video demo, it handles three real developer tasks back to back, fully offline.

QVAC

11,382,936 просмотров • 4 месяцев назад

STOP GUESSING WHICH LOCAL AI MODELS YOUR COMPUTER CAN RUN. LLMFIT SCANS YOUR HARDWARE AND TELLS YOU WHICH MODELS WILL ACTUALLY RUN WELL, INCLUDING THE BEST QUANTIZATION TO USE.
0:46

STOP GUESSING WHICH LOCAL AI MODELS YOUR COMPUTER CAN RUN. LLMFIT SCANS YOUR HARDWARE AND TELLS YOU WHICH MODELS WILL ACTUALLY RUN WELL, INCLUDING THE BEST QUANTIZATION TO USE.

0xMarioNawfal

48,706 просмотров • 1 месяц назад