Загрузка видео...

Не удалось загрузить видео

На главную

Factory benchmarks let you build your own model bench using your past coding agent runs. - Mirrors your environment, secrets, and MCPs for each run - Scores output on judging criteria you define - Generates a report with model recs using cost vs. quality Here's how it works:

warpdotdev's profile picture

Warp

58,705 subscribers

71,982 просмотров • 4 дней назад •via X (Twitter)

Наука и технологии
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at
2:33

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

132,403 просмотров • 25 дней назад

Create your own local and free AI agent using an open source model You can combine: - IBM's Granite 3.3 8B AI model - LM Studio to run it on a laptop - Smolagents to build your agent Small AI models are now powerful enough to run an autonomous agent. Thanks to IBM France for sponsoring this content!
8:44

Create your own local and free AI agent using an open source model You can combine: - IBM's Granite 3.3 8B AI model - LM Studio to run it on a laptop - Smolagents to build your agent Small AI models are now powerful enough to run an autonomous agent. Thanks to IBM France for sponsoring this content!

Paul Couvert

18,890 просмотров • 1 год назад

LM Studio is the most popular way to run open-source LLMs on your own hardware. Your Hermes Agent now runs natively on LM Studio: auto-discovering your models, loading them on demand with the right context size, and using the right reasoning level for each model.
0:25

LM Studio is the most popular way to run open-source LLMs on your own hardware. Your Hermes Agent now runs natively on LM Studio: auto-discovering your models, loading them on demand with the right context size, and using the right reasoning level for each model.

Nous Research

186,253 просмотров • 4 месяцев назад

1. Factory An advanced AI coding tool: generates complex apps, docs, works well with large existing projects, has access to web search, MCPs, can run code on my local machine + UX is 🤌 . best for coders. I'm building "Inbox agent" using this tool, gonna report back soon
0:25

1. Factory An advanced AI coding tool: generates complex apps, docs, works well with large existing projects, has access to web search, MCPs, can run code on my local machine + UX is 🤌 . best for coders. I'm building "Inbox agent" using this tool, gonna report back soon

John Rush

96,653 просмотров • 1 год назад

Run your coding agent on a local model, with QVAC. OpenCode now plugs into a model on your own machine through our official plugin. That changes two things. Privacy by default. Your repo, your prompts, and every command the agent runs stay on your device. No third-party model ever sees your code. For an agent that reads your codebase and runs commands, that is the point, not a nice-to-have. New workflows. With no meter and no rate limit, the agent does not have to stop. Let it loop on a task, refactor a whole repo overnight, or run a watch-build-fix loop that keeps going. Setup takes a few minutes: one line in opencode.json and OpenCode runs on a local QVAC model. In the video demo, it handles three real developer tasks back to back, fully offline.
1:53

Run your coding agent on a local model, with QVAC. OpenCode now plugs into a model on your own machine through our official plugin. That changes two things. Privacy by default. Your repo, your prompts, and every command the agent runs stay on your device. No third-party model ever sees your code. For an agent that reads your codebase and runs commands, that is the point, not a nice-to-have. New workflows. With no meter and no rate limit, the agent does not have to stop. Let it loop on a task, refactor a whole repo overnight, or run a watch-build-fix loop that keeps going. Setup takes a few minutes: one line in opencode.json and OpenCode runs on a local QVAC model. In the video demo, it handles three real developer tasks back to back, fully offline.

QVAC

11,382,936 просмотров • 2 месяцев назад

Stop using your agent logs just for debugging. Use them to train your own model. Today we're launching Agnost AI (YC S26)'s first model: agnost-*******-0.1 Trained for our first customer on their existing production traces & it beat the frontier: +22.9% task success −90.2% latency −94.5% cost If you ever thought of fine-tuning your own model, we'll do it for you, talk to us!
0:43

Stop using your agent logs just for debugging. Use them to train your own model. Today we're launching Agnost AI (YC S26)'s first model: agnost-*******-0.1 Trained for our first customer on their existing production traces & it beat the frontier: +22.9% task success −90.2% latency −94.5% cost If you ever thought of fine-tuning your own model, we'll do it for you, talk to us!

shubham

692,821 просмотров • 18 дней назад

If you want frontier-level coding and agent performance but you don't want to pay closed-model prices, GLM 5.2 is the open model you've probably been hearing about. Here's why you should run it on Fireworks. Your turn:
2:37

If you want frontier-level coding and agent performance but you don't want to pay closed-model prices, GLM 5.2 is the open model you've probably been hearing about. Here's why you should run it on Fireworks. Your turn:

Fireworks

145,611 просмотров • 2 месяцев назад

Introducing Arena Mode in Windsurf: One prompt. Two models. Your vote. Benchmarks don't reflect real-world coding quality. The best model for you depends on your codebase and stack. So we made real-world coding the benchmark. Free for the next week. May the best model win.
0:33

Introducing Arena Mode in Windsurf: One prompt. Two models. Your vote. Benchmarks don't reflect real-world coding quality. The best model for you depends on your codebase and stack. So we made real-world coding the benchmark. Free for the next week. May the best model win.

Devin Desktop

1,063,786 просмотров • 7 месяцев назад

Introducing Merge Gateway - Build Your Own Router. You're three sprints into your coding assistant. You pick the most hyped model, integrate, test, deploy. A month later, a new model drops. Now you re-test, re-integrate, re-deploy. Your product didn't change, but the benchmark did. That's how most AI teams operate. Chasing a "best" defined by people who've never seen their product. There is no best model. There's only the right one for your product, users, and use-cases. Build Your Own Router runs on your definition of good. Pick your benchmarks, weigh them, add your own evals. Merge routes every request to your winner. 👉$100 in credits to the first 200 people that comment
1:57

Introducing Merge Gateway - Build Your Own Router. You're three sprints into your coding assistant. You pick the most hyped model, integrate, test, deploy. A month later, a new model drops. Now you re-test, re-integrate, re-deploy. Your product didn't change, but the benchmark did. That's how most AI teams operate. Chasing a "best" defined by people who've never seen their product. There is no best model. There's only the right one for your product, users, and use-cases. Build Your Own Router runs on your definition of good. Pick your benchmarks, weigh them, add your own evals. Merge routes every request to your winner. 👉$100 in credits to the first 200 people that comment

Shensi Ding

3,667,560 просмотров • 3 месяцев назад

Introducing Warp Factories: open, flexible infrastructure for building cloud software factories. - Configure your factory as code - Use any model and any harness - Measure quality with evals and benchmarks on your own data - Built-in self-improvement and memory
8:56

Introducing Warp Factories: open, flexible infrastructure for building cloud software factories. - Configure your factory as code - Use any model and any harness - Measure quality with evals and benchmarks on your own data - Built-in self-improvement and memory

Warp

301,288 просмотров • 20 дней назад

Sub-agent Model Selection — Different Tasks, Different Models Your main agent runs Qwen3.6-Plus for quality. But not every subtask needs a flagship model. Now sub-agents can use a different model. Create a skill file with model: openai:qwen3.5-plus and the sub-agent runs on that model. Powerful model for the hard parts, fast model for the easy parts. Save tokens without sacrificing quality on what matters.
1:03

Sub-agent Model Selection — Different Tasks, Different Models Your main agent runs Qwen3.6-Plus for quality. But not every subtask needs a flagship model. Now sub-agents can use a different model. Create a skill file with model: openai:qwen3.5-plus and the sub-agent runs on that model. Powerful model for the hard parts, fast model for the easy parts. Save tokens without sacrificing quality on what matters.

Qwen

21,333 просмотров • 5 месяцев назад

This is crazy... Have you ever thought about the combination of AI and podcast? Now you can build your own AI podcast agent. Here's how it works:
0:38

This is crazy... Have you ever thought about the combination of AI and podcast? Now you can build your own AI podcast agent. Here's how it works:

Angry Tom

25,328 просмотров • 1 год назад

I made Apple's App Store is obsolete. I built an Claude Code for your iPhone, so you can build and run whatever app you want all on your device! Just describe an app and it writes the Swift, runs it, reads its own errors, and fixes them. Then it lands on your app's home screen. Here's exactly how it works 👇
0:48

I made Apple's App Store is obsolete. I built an Claude Code for your iPhone, so you can build and run whatever app you want all on your device! Just describe an app and it writes the Swift, runs it, reads its own errors, and fixes them. Then it lands on your app's home screen. Here's exactly how it works 👇

JC Builds

34,514 просмотров • 2 месяцев назад

You can legit build your own OpenClaw🦞 from scratch in about 20 minutes with the GitHub Copilot SDK. And when you build it, you can make it do anything. Here's how to build a personal AI assistant, and how I'm using one today.
21:57

You can legit build your own OpenClaw🦞 from scratch in about 20 minutes with the GitHub Copilot SDK. And when you build it, you can make it do anything. Here's how to build a personal AI assistant, and how I'm using one today.

Burke Holland

26,933 просмотров • 5 месяцев назад

here's how you can turn your claude into a fully autonomous AI agent that runs locally be smart and let it work for you while you sleep
3:03

here's how you can turn your claude into a fully autonomous AI agent that runs locally be smart and let it work for you while you sleep

Nidhi Singh

77,009 просмотров • 6 месяцев назад

Your perfect model is always ready. With Fotor, you can create your own AI model and use it to bring your marketing ideas to life. Generate consistent photos, videos, and campaign assets without the time and cost of traditional photoshoots. One AI model. Endless content. Create your model. Build your brand. Market smarter. Try Fotor AI Today！
0:31

Your perfect model is always ready. With Fotor, you can create your own AI model and use it to bring your marketing ideas to life. Generate consistent photos, videos, and campaign assets without the time and cost of traditional photoshoots. One AI model. Endless content. Create your model. Build your brand. Market smarter. Try Fotor AI Today！

Fotor

45,821 просмотров • 1 месяц назад

x402 SEO Agent you pass it a URL, it analyzes your site against top-ranking competitors and generates an actionable SEO strategy Here's how it works 🧵👇
0:43

x402 SEO Agent you pass it a URL, it analyzes your site against top-ranking competitors and generates an actionable SEO strategy Here's how it works 🧵👇

Ash

18,220 просмотров • 6 месяцев назад

Last week we launched Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency Watch our new video walkthrough to see how to build and run your own benchmark with Optima, and find the best model for your use case Start building your benchmark today at
17:43

Last week we launched Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency Watch our new video walkthrough to see how to build and run your own benchmark with Optima, and find the best model for your use case Start building your benchmark today at

Artificial Analysis

19,837 просмотров • 20 дней назад

Okay it works!!!! Now making the interface for it, I named it [ 👋 Hold a product ] You upload your product photo, you take a photo and it generates one with your AI model holding it Then from there you can press [ Talking video ], write a script and your AI model will present the product for you The AI character is from @lucataco93's tweets 😊
0:20

Okay it works!!!! Now making the interface for it, I named it [ 👋 Hold a product ] You upload your product photo, you take a photo and it generates one with your AI model holding it Then from there you can press [ Talking video ], write a script and your AI model will present the product for you The AI character is from @lucataco93's tweets 😊

@levelsio

188,528 просмотров • 1 год назад

🚀 Creating Your Own AI Agent Has Never Been This Easy! 🤖 Here’s a full walkthrough of building, customizing, and deploying your AI agent in minutes using AI Agent Factory. ✅ No-code tools ✅ NFT-based ownership ✅ Monetize your agent ✅ Run it on decentralized compute Whether you're a creator, a dev, or just AI-curious, this platform puts the power in your hands. Watch how it all works 👇
0:42

🚀 Creating Your Own AI Agent Has Never Been This Easy! 🤖 Here’s a full walkthrough of building, customizing, and deploying your AI agent in minutes using AI Agent Factory. ✅ No-code tools ✅ NFT-based ownership ✅ Monetize your agent ✅ Run it on decentralized compute Whether you're a creator, a dev, or just AI-curious, this platform puts the power in your hands. Watch how it all works 👇

AI Agent Factory

11,191 просмотров • 1 год назад