正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Introducing Vals-Smith: turn your code base into a customized benchmark. Public benchmarks tell you which model is strongest overall, not which model is the best on your code. Vals-Smith turns your merged pull requests into real coding tasks and measures the percentage a model can actually resolve. New models... show more

Vals AI

28,754 subscribers

46,847 次观看 • 3 个月前 •via X (Twitter)

科学技术教育健康养生

Anya Rossi• Live Now

Private livecam show

28 条评论

Vals AI 的头像

Vals AI3 个月前

Join the waitlist today!

Rayan Krishnan 的头像

Rayan Krishnan3 个月前

props to @OfirPress @jyangballin @KLieret for laying a lot of the ground work. by default we evaluate all models with mini swe agent, methodology is heavily swe bench inspired

Jerry Liu 的头像

Jerry Liu3 个月前

congrats on the launch!!

Gabriel Stengel 的头像

Gabriel Stengel3 个月前

Congrats @RayanKrishnan on the launch!

Rez Havaei 的头像

Rez Havaei3 个月前

Great work!

Aryo Patel 的头像

Aryo Patel3 个月前

Awesome release -- can't wait to try it out code speed about to double

Jihoon 的头像

Jihoon3 个月前

i had this idea for months and these days its great, you just have to wait because good ideas end up getting implemented thanks!

Braeden Norris 的头像

Braeden Norris3 个月前

Super cool launch, congrats!

Inflectiv AI ⧉ 的头像

Inflectiv AI ⧉3 个月前

🔥🔥

安叫兽|Bird🕊️ 🔶 BNB 的头像

安叫兽|Bird🕊️ 🔶 BNB3 个月前

拿自己仓库测，比看榜单踏实多了

Ronak Malde 的头像

Ronak Malde3 个月前

This is very exciting!!

icosaedro 的头像

icosaedro3 个月前

PRs as the benchmark set is the only metric that matters. Generic evals lie.

Sumer Sao 的头像

Sumer Sao3 个月前

congrats guys!

Jason Zhao 的头像

Jason Zhao3 个月前

Awesome stuff

Rona Wang 的头像

Rona Wang3 个月前

🚀

Vivek Gopalan 的头像

Vivek Gopalan3 个月前

Really important

Moritz Stephan 的头像

Moritz Stephan3 个月前

this is awesome

Daniel Rabinovich 的头像

Daniel Rabinovich3 个月前

This is really cool!

Faraz 的头像

Faraz3 个月前

Really cool work @RayanKrishnan !!

Dhruv Mangtani 的头像

Dhruv Mangtani3 个月前

sounds expensive lol

Sergey Karayev 的头像

Sergey Karayev3 个月前

repo-specific evals are the way. we built this into @Superconductor too, and you can run your own on merged PRs today no waitlist:

Dhruvik Parikh 🌎 的头像

Dhruvik Parikh 🌎3 个月前

🔥🔥🔥

shyamal 的头像

shyamal3 个月前

congrats on the launch!

iTokens.io ⚡️ AI API 的头像

iTokens.io ⚡️ AI API3 个月前

This feels like the direction coding evaluation is heading. The most useful benchmark isn't the one that ranks models globally—it's the one built from your own workflows and codebase.

Jihoon 的头像

Jihoon3 个月前

@grok competitor landscape in both closed and open source

RAZA | AI EXPLORER 的头像

RAZA | AI EXPLORER3 个月前

Measuring performance on your own code is a game-changer. Love this.

Lucas Campa 的头像

Lucas Campa3 个月前

Looks awesome 🚀🚀

Panizzutti 的头像

Panizzutti3 个月前

I like this but vibe coded video

相关视频

Introducing Arena Mode in Windsurf: One prompt. Two models. Your vote. Benchmarks don't reflect real-world coding quality. The best model for you depends on your codebase and stack. So we made real-world coding the benchmark. Free for the next week. May the best model win.

Introducing Arena Mode in Windsurf: One prompt. Two models. Your vote. Benchmarks don't reflect real-world coding quality. The best model for you depends on your codebase and stack. So we made real-world coding the benchmark. Free for the next week. May the best model win.

Devin Developers

1,073,143 次观看 • 8 个月前

🧳 Bring Your Own AI Models to GitHub Copilot in VS Code Use your own Azure-hosted AI models with GitHub Copilot in VS Code. See how Bring Your Own Key (BYOK) lets you connect models from Microsoft Foundry, choose which models appear in Copilot, and use the right model for different coding tasks.

🧳 Bring Your Own AI Models to GitHub Copilot in VS Code Use your own Azure-hosted AI models with GitHub Copilot in VS Code. See how Bring Your Own Key (BYOK) lets you connect models from Microsoft Foundry, choose which models appear in Copilot, and use the right model for different coding tasks.

Visual Studio Code

38,536 次观看 • 1 个月前

They're still your ideas. [code]smith just writes the code. [code]smith is a cloud software engineer that works alongside you. Autofixing PRs, CI failures and reviews.

They're still your ideas. [code]smith just writes the code. [code]smith is a cloud software engineer that works alongside you. Autofixing PRs, CI failures and reviews.

Blacksmith

3,621,539 次观看 • 2 个月前

SOMEONE MADE IT SO YOU CAN CODE WITH 174 AI MODELS FROM 23 PROVIDERS FOR FREE one npm package and one install: "npm i -g free-coding-models" switch between any model instantly. benchmark them against each other in real time to see which one actually codes best for your use case. no subscriptions. no API keys. no cost. 174 models. 23 providers, all free, all from your terminal. if you're tired of paying for 3 different AI subscriptions just to compare which model codes better, this is it

SOMEONE MADE IT SO YOU CAN CODE WITH 174 AI MODELS FROM 23 PROVIDERS FOR FREE one npm package and one install: "npm i -g free-coding-models" switch between any model instantly. benchmark them against each other in real time to see which one actually codes best for your use case. no subscriptions. no API keys. no cost. 174 models. 23 providers, all free, all from your terminal. if you're tired of paying for 3 different AI subscriptions just to compare which model codes better, this is it

Om Patel

93,660 次观看 • 6 个月前

Introducing Shiro - the CUTEST ai coding agent which turns your prompts into pull requests. Link in the comments. @JoinShiro

Introducing Shiro - the CUTEST ai coding agent which turns your prompts into pull requests. Link in the comments. @JoinShiro

Dev

496,344 次观看 • 1 年前

This approach has made Sonnet the model of choice for developers worldwide. In addition to our new model, we're launching Claude Code, our first coding tool, in a limited research preview. With Claude Code, you can delegate substantial tasks to Claude—right from your terminal.

This approach has made Sonnet the model of choice for developers worldwide. In addition to our new model, we're launching Claude Code, our first coding tool, in a limited research preview. With Claude Code, you can delegate substantial tasks to Claude—right from your terminal.

Anthropic

1,141,101 次观看 • 1 年前

Introducing Claude Code on the web. You can now delegate coding tasks to Claude without opening your terminal.

Introducing Claude Code on the web. You can now delegate coding tasks to Claude without opening your terminal.

Claude

2,666,443 次观看 • 1 年前

Meet AutoRouter, the most intelligent way to pick models You’re wasting tokens selecting frontier models for every prompt, and sometimes for worse performance AutoRouter handles the decision layer: - Understands the prompt request - Picks the best model fit for the task - Saves you money at every turn - Plugs into your current setup Works with your plans & every leading model inside Codex/Claude Code Beta is live:

Meet AutoRouter, the most intelligent way to pick models You’re wasting tokens selecting frontier models for every prompt, and sometimes for worse performance AutoRouter handles the decision layer: - Understands the prompt request - Picks the best model fit for the task - Saves you money at every turn - Plugs into your current setup Works with your plans & every leading model inside Codex/Claude Code Beta is live:

ty

82,364 次观看 • 2 天前

🎙️ Dictate in Multiple Languages in #VSCode Dictation can detect your language and turn your speech into text using a local model. You can use it in Chat, the editor, and the integrated terminal.

🎙️ Dictate in Multiple Languages in #VSCode Dictation can detect your language and turn your speech into text using a local model. You can use it in Chat, the editor, and the integrated terminal.

Visual Studio Code

27,733 次观看 • 2 个月前

Multimodal AI is here 🤯 GPT-4 can now turn your images into a text file in a snap with the new code interpreter model. Witness the OCR magic in action 🔥

Multimodal AI is here 🤯 GPT-4 can now turn your images into a text file in a snap with the new code interpreter model. Witness the OCR magic in action 🔥

Shubham Saboo

727,673 次观看 • 3 年前

Introducing OpenResearch: run parallel research agents with any model Bring your own compute and your code + data stays with you A virtual lab on your machine

Introducing OpenResearch: run parallel research agents with any model Bring your own compute and your code + data stays with you A virtual lab on your machine

alphaXiv

32,864 次观看 • 3 个月前

Don't talk directly to a model. You can make your application 10x more flexible by simply adding one intermediate layer between your code and the model. This is the simplest trick you can use to improve the architecture of whatever you are building.

Don't talk directly to a model. You can make your application 10x more flexible by simply adding one intermediate layer between your code and the model. This is the simplest trick you can use to improve the architecture of whatever you are building.

Santiago

88,884 次观看 • 6 个月前

Introducing Merge Gateway - Build Your Own Router. You're three sprints into your coding assistant. You pick the most hyped model, integrate, test, deploy. A month later, a new model drops. Now you re-test, re-integrate, re-deploy. Your product didn't change, but the benchmark did. That's how most AI teams operate. Chasing a "best" defined by people who've never seen their product. There is no best model. There's only the right one for your product, users, and use-cases. Build Your Own Router runs on your definition of good. Pick your benchmarks, weigh them, add your own evals. Merge routes every request to your winner. 👉$100 in credits to the first 200 people that comment

Introducing Merge Gateway - Build Your Own Router. You're three sprints into your coding assistant. You pick the most hyped model, integrate, test, deploy. A month later, a new model drops. Now you re-test, re-integrate, re-deploy. Your product didn't change, but the benchmark did. That's how most AI teams operate. Chasing a "best" defined by people who've never seen their product. There is no best model. There's only the right one for your product, users, and use-cases. Build Your Own Router runs on your definition of good. Pick your benchmarks, weigh them, add your own evals. Merge routes every request to your winner. 👉$100 in credits to the first 200 people that comment

Shensi Ding

3,668,527 次观看 • 5 个月前

Introducing WrapdCode Claude Code with Bring Your Own Model 1. Use GPT, Gemini, Llama, Mistral - 300+ models 2. Same Claude Code UI, zero changes 3. Plug in your API key and go Stop being stuck on one model.

Introducing WrapdCode Claude Code with Bring Your Own Model 1. Use GPT, Gemini, Llama, Mistral - 300+ models 2. Same Claude Code UI, zero changes 3. Plug in your API key and go Stop being stuck on one model.

aditya

29,014 次观看 • 6 个月前

Fusion Models by robinhood:0xa92768863a55d8a0591709f7f5e594a249d36ea3, explained properly. Why a panel beats one model: Ask three models the same hard question and they disagree. The disagreement is the information, where they contradict each other is where one of them is wrong, and a judge can see it. OpenRouter shipped exactly this in June and their panels beat any single frontier model. Credit to them and their $10B sale to stripe. Why the same panel for everyone is the wrong shape: You don't have one best model. You have a best model per job. The one you trust for code is not the one you trust for writing, and neither is the one you'd pick for a five-line reply. A panel chosen per request, identical for every user, can't know that. A panel arranged around you can. The number that matters isn't price: It's cost per answer that works. Model A costs $4 per million tokens and gets your job right one time in four. Model B costs $10 and gets it right three times in four. Per answer you actually keep, A costs $16 and B costs $13. The cheap model was the expensive one. Everyone comparing price lists is measuring the wrong thing. You never rate anything: Stopping an answer halfway is a rating. Regenerating is a rating. Switching model and asking again is the loudest rating there is. Every one of those already writes a row we can learn from. The router learns which models earn your credits, per job, and tries a cheaper one now and then to check. Fusion only when it pays: Cost mode tries the cheap model first and escalates only if a judge says it failed. Performance mode fans out to your panel and merges. Most simple prompts never fuse. That's why it costs less, not more. Put it together and you have a personal model: Not trained. Assembled. Your profile from a two-minute questionnaire and everything you've imported from Claude or ChatGPT. Your memory, shared across every model so switching never starts from zero. Your skills, workflows you build once and run on any model. Your scorecard, which models earn your credits per job. One per person. Nobody else has yours. Every model is someone else's; the arrangement is you, and it leaves with you. People aren't loyal to models. They're loyal to results. A moat can't be a commodity, but it can be our own creation.

askr

13,900 次观看 • 1 个月前

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

133,793 次观看 • 2 个月前

Introducing Cursor 2.0. Our first coding model and the best way to code with agents.

Introducing Cursor 2.0. Our first coding model and the best way to code with agents.

Cursor

3,572,368 次观看 • 1 年前

Stop hardcoding one model name in your code. Now you can give each request a policy: a small rule for what the call needs It picks the right model for the job, on your own keys This is unhardcoded, our new open source routing for AI models, live today

Stop hardcoding one model name in your code. Now you can give each request a policy: a small rule for what the call needs It picks the right model for the job, on your own keys This is unhardcoded, our new open source routing for AI models, live today

GenLayer

98,139 次观看 • 4 个月前

Run your coding agent on a local model, with QVAC. OpenCode now plugs into a model on your own machine through our official plugin. That changes two things. Privacy by default. Your repo, your prompts, and every command the agent runs stay on your device. No third-party model ever sees your code. For an agent that reads your codebase and runs commands, that is the point, not a nice-to-have. New workflows. With no meter and no rate limit, the agent does not have to stop. Let it loop on a task, refactor a whole repo overnight, or run a watch-build-fix loop that keeps going. Setup takes a few minutes: one line in opencode.json and OpenCode runs on a local QVAC model. In the video demo, it handles three real developer tasks back to back, fully offline.

Run your coding agent on a local model, with QVAC. OpenCode now plugs into a model on your own machine through our official plugin. That changes two things. Privacy by default. Your repo, your prompts, and every command the agent runs stay on your device. No third-party model ever sees your code. For an agent that reads your codebase and runs commands, that is the point, not a nice-to-have. New workflows. With no meter and no rate limit, the agent does not have to stop. Let it loop on a task, refactor a whole repo overnight, or run a watch-build-fix loop that keeps going. Setup takes a few minutes: one line in opencode.json and OpenCode runs on a local QVAC model. In the video demo, it handles three real developer tasks back to back, fully offline.

QVAC

11,382,936 次观看 • 4 个月前

STOP GUESSING WHICH LOCAL AI MODELS YOUR COMPUTER CAN RUN. LLMFIT SCANS YOUR HARDWARE AND TELLS YOU WHICH MODELS WILL ACTUALLY RUN WELL, INCLUDING THE BEST QUANTIZATION TO USE.

STOP GUESSING WHICH LOCAL AI MODELS YOUR COMPUTER CAN RUN. LLMFIT SCANS YOUR HARDWARE AND TELLS YOU WHICH MODELS WILL ACTUALLY RUN WELL, INCLUDING THE BEST QUANTIZATION TO USE.

0xMarioNawfal

48,706 次观看 • 1 个月前