introducing OpenBench v1, an open framework for measuring AI...

Uploaded: 2026-07-21T16:32:16.000Z
Duration: PT20.714S
Channel: Matthew Lam

0:46

Anthropic Technical Staff Jess Yan says the harness and the model cannot be pulled apart without giving up performance, though she flags her own bias. "I think I am quite biased but I also think that it is impossible to get the maximum possible performance without tying together the harness and the model." "Now the components of the harness and maybe the thickness of the harness will change over time as models get more and more capable." "However, when we test our models and when we are assessing their performance we always have to test it in conjunction with a harness." "And are we going to test it with all the different harnesses of the world? We're going to select the harnesses that we have built." "And so there is an aspect of the necessity of building models is that you have to be testing them with harnesses, and that sort of keeps them paired together."

Fireside Alpha

584,074 views • 1 month ago

0:35

this is huge news from Cursor, they've pulled of the impossible and turned their ai-wrapper into a in-destructible moat sam altman literally called it cursor's ai agent harness is available for anyone to build on, which means ai models are now a commodity > 1-time install and now anyone can run cursor's agent locally or via cloud. use any model (e.g. gpt 5.5) but with the added cursor harness that makes it 10X better. > its so good that 3 of cursors biggest competitors are embedding it into their products. > now cursor DOESN'T DEPEND on anthropic or openai. their own model (composer 2) competes directly! yesterday sam altman said the ai model and harness are one and the same and today cursor turned their harness into a self-owned moat fucking masterclass (coming from a former cursor / ai wrapper hater)

Ejaaz

950,452 views • 4 months ago

2:33

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

132,338 views • 24 days ago

0:47

When I talk to serious software engineers, they tell me everyone is sleeping on Codex + GPT 5.2 from OpenAI Which is why I was hyped to get an insiders "how to" from Alexander Embiricos product lead for Codex, showing us how he and the team use Codex to ship almost everything at oai. He'll show us how to: - jump into a new repo and start contributing, even as a non-technical PM - run git worktrees to parallelize agent tasks across codex instances - use best practices and to spec out your work and let codex rip We also talk model vs. harness and how to spy on codex development (which is all OSS!) to understand how to optimize your own implementations. This episode is brought to you by our amazing sponsors 🧠🤑 Brex - The intelligent finance platform built for founders 🦎🐛 Greptile - Your AI code review platform Watch it here:

claire vo 🖤

48,102 views • 7 months ago

1:54

How to use 50+ API keys (models) for FREE on OpenClaw API??? - go to - login or register your account - click on "more models" - click on "use case" and select what you need it for - choose the model and open it - click on "view code" → "Generate API key" many models don't allow direct deploy, so use the "view code" button to generate API access basically Nvidia NIM gives you the ability to test almost any model from their list for FREE some of them are not worse than GPT 5.2 or Claude Opus 4.6, some might even perform better depending on the task how to understand if a model is efficient and compare it with others??? - go to - type the model name in search - click on "benchmarks" - you’ll see performance tests and rankings this way you can easily compare free models with paid ones of course there are RPM limits, on many models it’s around ~40 requests per minute each model is different, after generating the API key, RPM limits are shown in the top-right corner nothing stops you from using them, many work perfectly fine, super solid option for first tests and for learning OpenClaw or any other system where you need an AI API model

Ronin

58,974 views • 6 months ago

0:42

I use subagents for ~80% of my Codex tasks these days. If you're new to subagents, here's a quick primer: - Subagents = separate agents that work on a specific piece of work in a fresh isolated context window. - Your main thread can create subagents and delegate work to them. - Subagents report back when tasks are complete. - Codex subagents are especially useful for read-heavy tasks (e.g research, investigation, and review). - Also useful for testing and building parts of a project in parallel. Subagents are great for working on focused parts of a big task, like planning, implementation, testing, or review. My biggest pro tip for using subagents in Codex: You don’t have to design the entire delegation plan yourself. Just ask Codex to “Use subagents as needed,” and then Codex can then decide where subagents would help it in the task. Here's a demo of using subagents in Codex:

Avthar

18,223 views • 7 days ago

0:23

Tune Studio is an end-to-end platform for developing applications using Large Language Models. So far, I haven't seen any other platform like this one. You can do everything here: 1. You can curate your data. 2. Use the playground to play with different models and try your ideas. 3. Fine-tune an open-source model on your data. 4. Deploy the model when you are done. This is awesome for anyone building generative AI applications. You can use Tune Studio to work with any of the open-source models out there. They were one of the few companies to host Llama 2 and Llama 3 before anyone else. Here is a link to check it out: One of their main selling points is that Tune Studio scales! You don't have to worry about serving your model to lots of users. They also have built-in user management, authentication, on-prem support, user context management, and pretty much everything you need to build generative AI applications. Thanks to the Tune team for collaborating with me on this post. We are living through the best years of development tools for AI developers. The field is unstoppable.

Santiago

39,101 views • 2 years ago

6:05

You can now fine-tune Llama 3 without writing a single line of code! We are moving at breakneck speed. I recorded a video to show you how to fine-tune any open-source model in a few minutes. I'm using a GPT capable of taking a problem and turning it into a fine-tuned model that will solve it. You don't have to write any code. You only need to explain to a GPT what problem you want to solve and tell it you want to use Llama 3. For example, "fine-tune Llama 3" or "deploy zephyr." It feels magic. The system will recommend a dataset and fine-tune the model for you. I'm using Monster API, a platform that specializes in making fine-tuning and deploying open-source models easy and fast. Their stack is well-optimized to maximize fine-tuning efficiency using techniques like Q-Lora and vLLM. They are behind the GPT. Here is what you need to do: 1. Create an account at 2. Load the GPT with the link below This is as simple as it gets. When you are done, you can click a button to deploy the model and start using it. I have 10,000 free credits for anyone using the code "SANTIAGO" in the dashboard. You can use these credits to access, fine-tune, and deploy these open-source models. You can also keep up with their latest updates, and get free credits and special offers on their Discord server:

Santiago

324,621 views • 2 years ago

1:56

Our first short course with Anthropic! Building Towards Computer Use with Anthropic. This teaches you to build an LLM-based agent that uses a computer interface by generating mouse clicks and keystrokes. Computer Use is an important, emerging capability for LLMs that will let AI agents do many more tasks than were possible before, since it lets them interact with interfaces designed for humans to use, rather than only tools that provide explicit API access. I hope you will enjoy learning about it! This course is taught by Anthropic's Head of Curriculum, Colt_Steele. You'll learn to apply image reasoning and tool use to "use" a computer as follows: a model processes an image of the screen, analyzes it to understand what's going on, and navigates the computer via mouse clicks and keystrokes. This course goes through the key building blocks, and culminates in a demo of an AI assistant that uses a web browser to search for a research paper, downloads the PDF, and finally summarizes the paper for you. In detail, you’ll: - Learn about Anthropic's family of models, when to use which one, and make API requests to Claude - Use multi-modal prompts that combine text and image content blocks, and also work with streaming responses - Improve your prompting by using prompt templates, using XML to structure prompts, and providing examples - Implement prompt caching to reduce cost and latency - Apply tool-use to build a chatbot that can call different tools to respond to queries - See all these building blocks come together in Computer Use demo Please sign up here:

Andrew Ng

170,541 views • 1 year ago

2:55

People are overlooking Google Gemini Realtime models for computer use It gave me sub 100ms latency with computer use. It has a much larger context window and is much cheaper as well Combine that with local OCR and local screen detection model based on Omniparser by Microsoft it works under 100ms action taking when combined with Cua I also put in a harness for Nous Research Hermes with it. You can access it all at your tip of your cursor. You can draw on your screen to give your agents a context And I am making it Open Source! Link in the Comments Sundar Pichai Min-Liang Tan Mojtaba Seyedhosseini

Milind S

38,361 views • 3 months ago

23:56

An agent is three things: a harness, a model, and context. If you're serious about owning your intelligence, you probably want to own all three. LangChain founder Harrison Chase joined us at our Sequoia Capital Own Your Intelligence to talk about the piece that often gets the least attention: the harness. He offers a clear heuristic for when to build your own. The more out of distribution you are from what the models were trained on, the more you'll want to customize. And good technical content on how to actually measure performance with evals and langsmith. 00:00 Introduction 00:58 The three parts of an agent: harness, model, context 02:12 What a harness actually does 03:25 Customizing the core loop with middleware 04:41 Sandboxes, file systems, sub-agents, summarization 05:47 Cognitive architectures — and when you still need them 07:03 Build your own harness or use off the shelf? 08:24 In-distribution vs. out-of-distribution: the file-editing example 09:39 Why evals define what "good" means in an organization 11:04 Harbor: what an eval task actually looks like 12:11 Comparing harnesses and models on accuracy, latency, and cost 13:20 Why observability is underrated — it's usually the context 14:34 The data flywheel: traces → curation → experiments 15:42 Getting feedback through UX design and online evaluators 16:51 Demo: LangSmith Engine 19:23 Q&A: Running Engine on Engine, and "codex-ification" 20:44 Q&A: Will harnesses converge or diverge?

Sonya Huang 🐥

77,519 views • 24 days ago

4:14

New short course: Prompt Engineering with Llama 2, built in collaboration with Meta AI at Meta, and taught by Amit Sangani! Meta's Llama 2 has been game-changing for AI. Building with open source lets you control your own data, scrutinize errors, update (or not) the models as you please, and work alongside the global community advancing open models. Llama isn't a single model, it's a collection of models. In this course, you'll: - Learn the differences between different Llama 2 flavors, and when to use each. - Prompt the Llama chat models -- you'll also see how Llama's instruction tags work -- so they can help you with day-to-day tasks, like writing or summarization. - Use advanced prompting, like few-shot prompting for classification, and chain-of-thought prompting for solving logic problems. - Use specialized models in the Llama collection for specific tasks, like Code Llama to help you write, analyze, and improve code, and Llama Guard, which checks prompts and model responses for harmful content. The course also touches on how to run Llama 2 locally on your own computer. I hope you’ll take this course and try out these powerful, open models!

Andrew Ng

162,875 views • 2 years ago

2:15

i love Grok Grok Bot i do not love its rate limits here is a simple way to get more usage out of it: install your coding agents on Grok Bot's virtual machine Codex, Cursor, Grok Build, Antigravity, etc then, install herdr to manage those CLI agent sessions now give Grok Bot a rule: use Herdr + CLI agents for anything more than conversational tasks your rate limits for Grok Bot will only be coordination and prompting, rather than the coding tasks themselves and you can use those Cursor and Grok Build limits AND crank on Grok Bot without getting rate limited as fast

Aaron Makelky

175,166 views • 8 days ago

0:25

We added a device tool bar to the Codex in-app browser, so it's easier to build and test responsive apps! Now, you can have Codex test your app in different dimensions, so it can fix bugs & improve UI for every device. Just click the 3 dots on the right of the URL bar to use it! As part of this change, we also: - improved the speed of browser use (~30% in my vibe testing) - tightened the animation of the cursor - added a way to hide the composer in full screen mode - fixed a few bugs for browser use in Windows Oliver Tepman and John Nastos cooked on this one! Try it out and let us know what you think!

James Sun

148,458 views • 4 months ago

1:47

Bash is all you need! Which is why I'm introducing my holiday project: just-bash just-bash is a pretty complete implementation of bash in TypeScript designed to be used as a bash tool by AI agents. Because it turns out agents love exploring data via shell scripts, even beyond coding. It comes with grep, sed, awk and the 99th percentile features that an agent like Claude Code or Cursor would use. In fact, Claude Code can use it for secure bash execution. In the package - A bash-tool for AI SDK - A binary for use by yourself or your coding agents - An overlay filesystem to feed files to your agent securely - A Vercel Sandbox compatible API, so you can quickly upgrade to a real VM if you need to run binaries - An example AI agent that explores the just-bash code base using just-bash - I imported the Oils shell bash compatibility suite and just-bash passes a very good chunk What is interesting about this codebase: It was essentially entirely written by Opus 4.5. Coding agents love bash and they are good at reproducing it. They are also great at text-book recursive descent parsers and AST tweet-walk interpreters. That said, it is, like, a lot of code and I didn't read it all 😅. This is very much a hack, but it also seems to be _really_ useful. I haven't really found anything agents want to use that it doesn't support and it's fast and secure (caveats apply). It doesn't have write access to your computer and the filesystem is given a root that the agent cannot escape from. Find it at Related: Our recent blog post how we migrated our data analysis agent to bash tools and achieved incredible quality improvements The video shows the example agent investigating the just-bash code base

Malte Ubl

125,326 views • 8 months ago

0:45

GPT-6 Astra “uses Unity to assemble a city scene from existing assets, allowing users to easily create an immersive 3D environment to explore that matches their vision.” “Alongside Astra, we are also updating the Codex harness to significantly improve the speed of computer use. Combined with Astra's efficiency, this translates to a 1.9x faster task completion compared to the current GPT-5.6 Sol experience, on the Mind2Web benchmark. “ “The model's improvements on speed mean it can take on many time consuming life tasks for you, faster than you can.*” I’ll admit, others and I have created far more impressive demos, but keep in mind this was done in Unity, which is a much more difficult environment for models to work with. If they weren’t using computer use which would’ve been extremely slow they were likely manipulating the project programmatically through C# and/or Unity’s YAML-serialized scene files which again models are pretty bad at. So this is a solid upgrade from 5.6 - from what I’ve seen unity!

Chris

22,147 views • 3 days ago

13:34

BREAKING: GPT-5.6 Sol is out—AND Codex has been merged into ChatGPT Desktop as ChatGPT Codex. This combo model and desktop app harness are the gold-standard for knowledge work in AI. 5.6 is powerful, fast, half the price of Fable, and my default for almost everything. We’ve been testing it internally Every 🪨 for about a month across coding, writing, design, and knowledge work. Here’s our day-zero vibe check: - An A-tier coder—but it’s not Fable. Sol scored 56/100 on our Senior Engineer benchmark compared to a 91 for Fable. I think the 56/100 undersells it, it's an excellent implementor, and very smart. But Fable just writes conceptually cleaner code and works better at the top end of task complexity. PRO-TIP: Use GPT-5.6 as Fable's subagent for the most goated combo in AI coding. - The best writer of the frontier models. It’s clearer and more concise than Fable or Opus 4.8, without the overexplaining or weird private language. It can one-shot marketing emails, help you workshop taglines, and explain complex concepts clearly. It's also super fast, which makes it easy to collaborate with. - Design is better, but not top-tier. It has noticeably more taste than 5.5, but Fable and Opus 4.8 are still playing at a different level. See examples in the video and vibe check below. - The real leap is knowledge work. Sol is the first model I’ve trusted to run whole loops of knowledge work—not just help with individual tasks. I use it to process email, surface decisions from meetings and Slack, find job candidates, scan Facebook Marketplace for furniture, and log my meals. It has shifted my job from doing the work to tending the system that does it. - The merged app is fine. I was extremely worried about this because I love the Codex app. OpenAI was caught in an interesting position: How to make an agent orchestration app for regular ChatGPT consumers, coders, and businesses all in one app. They now split the interface between ChatGPT Work and ChatGPT Codex. They're basically the same except Work hides code. And "Chat" has been demoted to 2nd tier status for quick questions in either one. It's not a big leap, but it's not a huge setback either. And it remains my favorite of the desktop agent orchestration apps. Verdict: If I really had to put my finger on it, I'd say Fable has way more big model smell. But that means it's a skill in itself to get value out of it—99% of people are still not there yet. GPT-5.6 is almost as powerful, but is easy to use, fast, and relatively cheap. It should give you an early sense of where model work is going. Full Every 🪨 Vibe Check:

Dan Shipper 📧

146,015 views • 1 month ago

3:07

OpenAI just announced API access to o1 (advanced reasoning model) yesterday. I'm delighted to announce today a new short course, Reasoning with o1, built with OpenAI, and taught by Colin Jarvis, Head of AI Solutions at OpenAI, to show you how to use this effectively! Unlike previous language models which generate output directly, o1 “thinks before it responds,” and generates many reasoning tokens before returning a more thoughtful and accurate response. It is great at complex reasoning -- including planning for agentic workflows, coding, and domain-specific reasoning in STEM fields like law. But how you should use it is quite different from other LLMs. I think o1 will be a game changer for many AI applications; and in this course, you'll learn how to use it effectively. In detail, you’ll: - Learn to recognize what tasks o1 is suited for, and when to use a smaller model, or combine o1 with a smaller model - Understand the new principles of prompting reasoning models: Be simple and direct; no explicit chain-of-thought required; use structure; show rather than tell - Implement multi-step orchestration in which o1 plans, and hands tasks over to gpt-4o-mini to execute specific steps; this illustrates a design pattern to optimize intelligence (accuracy) and cost - Use o1 for a coding task to build a new application, edit existing code, and test performance by running a coding competition between o1-mini and GPT 4o - Use o1 for image understanding and learn how it performs better with a "hierarchy of reasoning," in which it incurs the latency and cost upfront, preprocessing the image and indexing it with rich details so it can be used for Q&A later - Learn a technique called meta-prompting, in which you use o1 to improve your prompts. Using a customer support evaluation set, you'll iteratively use o1 to modify a prompt to improve performance You'll also learn about how OpenAI used reinforcement learning to produce a model that uses "test-time compute" to improve performance. I think you'll find this course enjoyable and valuable. Please sign up for it here:

Andrew Ng

357,661 views • 1 year ago

3:28

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on Hugging Face, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...

kwindla

325,963 views • 10 days ago

14:03

Here is EVERY AI tool you need to be using in August 2026 I've personally used each for 100+ If you master them and use them all, you'll be a productivity MACHINE The list of the tools are down below. How to use each is in the video! • Best planning model: Fable 5 medium- smartest model out there. Use it to do all your ideation and planning • Best execution model: ChatGPT 5.6 sol medium- best bang for your buck. TONS of usage and incredibly smart • Best AI tool: ChatGPT Voice: the future of interfacing with technology • Best AI agent harness: ChatGPT Desktop app- can do anything any other AI agent can do, but does it better • Best model for building UI/UX/games: Claude Code w/ Claude Opus 5- this model gets a lot of hate. Some of it warranted, most of it not. NOTHING beats Opus 5 for UI • Local model: Qwen 3.6 27b on DGX Spark- best speed and intelligence ratio in local AI • Other must have tools: Buzz from Jack Dorsey. The future of AI is multiplayer, and this is the first true multiplayer AI platform out there

Alex Finn

41,771 views • 27 days ago

Live Cam

Video Failed to Load

Matthew Lam

Anya Rossi• Live Now

0 Comments

Related Videos