Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

We just released the Databricks synthetic evaluation SDK! 🎉 We’ve found that synthesizing evals is a great way to hill climb your AI system before you’re able to get labels from domain experts. With this SDK, and our fancy diffing UI, users are able to improve the inner loop... show more

Nikhil Thorat

9,449 subscribers

10,446 views • 1 year ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

3 Comments

Hamel Husain1 year ago

@databricks Looks cool! Do you need to be using MLFlow to take advantage of this? BTW I didn't even know until now that MLFlow had LLM tools in it!!!

floating point1 year ago

@databricks I don't understand the workflow. So, say I see the agent made a wrong step, I see the diff and where. What is my next step? With agents I can't just add positive and negative demonstration to the training set, I have to debug the prompts, your tool does not solve that, right?

czverse.𝕏1 year ago

@databricks Game changer! Synth evals + diffing UI = faster, smarter iterations. Can't wait to try this out!

Related Videos

Grok3 Integration Now Available in AI Agent SDK! We’re thrilled to announce the integration of Grok3 into the AI Agent SDK, brining developers more AI capabilities! 🎉 🚀 Grok3 Integration: Easy integration with Grok3 for your projects. 🚀 AI Provider Enum: Quickly switch between multiple AI providers, including Grok3. 🔖 Check out the doc for the guide and more details: 🎬 Watch demos on setting up Grok3 and transacting via a Telegram powered by Grok3 with the SDK 👇 Dive in and start building with Grok3 today!

Grok3 Integration Now Available in AI Agent SDK! We’re thrilled to announce the integration of Grok3 into the AI Agent SDK, brining developers more AI capabilities! 🎉 🚀 Grok3 Integration: Easy integration with Grok3 for your projects. 🚀 AI Provider Enum: Quickly switch between multiple AI providers, including Grok3. 🔖 Check out the doc for the guide and more details: 🎬 Watch demos on setting up Grok3 and transacting via a Telegram powered by Grok3 with the SDK 👇 Dive in and start building with Grok3 today!

Crypto.com AI Agent

23,363 views • 1 year ago

Our AI Agent Launchpad is LIVE. Check out is a quick tutorial video. We are leveraging SuperHype 's Hyperliquid native token launchpad SDK and all AI Agent token will get bonded to SuperHype 's HyperEVM AMM DEX. A new era has begun for The Farm😈

Our AI Agent Launchpad is LIVE. Check out is a quick tutorial video. We are leveraging SuperHype 's Hyperliquid native token launchpad SDK and all AI Agent token will get bonded to SuperHype 's HyperEVM AMM DEX. A new era has begun for The Farm😈

thefarm.fun

27,383 views • 1 year ago

We worked with OpenAI to build a native integration for the OpenAI Agents SDK. Today, we are very excited to launch as a tracing processor. With just a few lines of code, you can trace all your agent workflows and debug them much faster. Check out this quick demo to get started:

We worked with OpenAI to build a native integration for the OpenAI Agents SDK. Today, we are very excited to launch as a tracing processor. With just a few lines of code, you can trace all your agent workflows and debug them much faster. Check out this quick demo to get started:

Respan

58,055 views • 1 year ago

🔥 We fine-tuned Gemma 3 using Unsloth AI on a custom instruction dataset scraped and built with . The model is now able to answer questions about OpenAI Agents SDK documentation. Check out the demo below👇

🔥 We fine-tuned Gemma 3 using Unsloth AI on a custom instruction dataset scraped and built with . The model is now able to answer questions about OpenAI Agents SDK documentation. Check out the demo below👇

Eric Ciarla (hiring)

28,794 views • 1 year ago

Next on ACP dev onboarding - in this walkthrough, John shows us how to test agents using the ACP SDK and Butler sandbox mode. This video covers: • Using our ACP SDK examples to test your agent • Testing your agent using Butler sandbox mode • How to get your agent graduated (and available for query via butler!) Useful links • ACP Node SDK: • ACP Python SDK:

Next on ACP dev onboarding - in this walkthrough, John shows us how to test agents using the ACP SDK and Butler sandbox mode. This video covers: • Using our ACP SDK examples to test your agent • Testing your agent using Butler sandbox mode • How to get your agent graduated (and available for query via butler!) Useful links • ACP Node SDK: • ACP Python SDK:

Celeste Ang

14,631 views • 5 months ago

Introducing /orchestrate, a skill that recursively spawns agents to tackle your most ambitious tasks with the Cursor SDK. We’ve used it to: - Autoresearch our internal skills, cutting token use by 20% while improving evals - Cut cold start times on our internal backend by 80%

Introducing /orchestrate, a skill that recursively spawns agents to tackle your most ambitious tasks with the Cursor SDK. We’ve used it to: - Autoresearch our internal skills, cutting token use by 20% while improving evals - Cut cold start times on our internal backend by 80%

Cursor

533,881 views • 2 months ago

A few months ago we set out to understand exactly how our models can be used to accelerate progress in robotics and other AI systems that don’t just represent the world, but are actually able to interact with it. We’ve been collaborating with leading companies in the space and we’ve developed an initial approach that we're now opening up to accelerate development across the industry. Our fifth announcement is GWM Robotics. GWM Robotics is a learned simulator that generates synthetic data for scalable robot training and policy evaluation, removing the bottlenecks of physical hardware.

A few months ago we set out to understand exactly how our models can be used to accelerate progress in robotics and other AI systems that don’t just represent the world, but are actually able to interact with it. We’ve been collaborating with leading companies in the space and we’ve developed an initial approach that we're now opening up to accelerate development across the industry. Our fifth announcement is GWM Robotics. GWM Robotics is a learned simulator that generates synthetic data for scalable robot training and policy evaluation, removing the bottlenecks of physical hardware.

Runway

35,040 views • 6 months ago

Now it’s time to onboard more builders. We’ve put together a short developer onboarding video. The video covers: - 1⃣: How to build your first agent with the Unibase Agent SDK and deploy it to Unibase AIP — from anywhere - 2⃣: How to call deployed agents and execute tasks via the Unibase AIP SDK — from anywhere Next up, we’ll show how to use Unibase Membase to persist the memory of the OpenClaw bot 🧠 🔗Useful Link ▫️Unibase Docs: ▫️Unibase Agent SDK: Your Fast way to spin up your first agent with AIP. ▫️Unibase AIP SDK & DOCS ▫️AIP 2.0 DOCS: *As the SDK is under active development, certain interfaces may be subject to change. Please refer to the latest GitHub source for the authoritative implementation.

Now it’s time to onboard more builders. We’ve put together a short developer onboarding video. The video covers: - 1⃣: How to build your first agent with the Unibase Agent SDK and deploy it to Unibase AIP — from anywhere - 2⃣: How to call deployed agents and execute tasks via the Unibase AIP SDK — from anywhere Next up, we’ll show how to use Unibase Membase to persist the memory of the OpenClaw bot 🧠 🔗Useful Link ▫️Unibase Docs: ▫️Unibase Agent SDK: Your Fast way to spin up your first agent with AIP. ▫️Unibase AIP SDK & DOCS ▫️AIP 2.0 DOCS: *As the SDK is under active development, certain interfaces may be subject to change. Please refer to the latest GitHub source for the authoritative implementation.

Unibase

19,712 views • 5 months ago

We've been obsessing a lot about creating the best possible AI support experience Dub.co. And I think we've found the magic formula. The stack: ✦ Mintlify → docs + help articles ✦ Upstash vector → vector search ✦ AI SDK + Claude sonnet → LLM/retrieval ✦ Plain → support infra Here's how it works: We consolidated our docs + help articles on Mintlify (best-in-class docs platform) so all our support content is now in one place. On every push to main, we sync changes to our Upstash Vector DB. We also built a custom chat interface that retrieves content from Upstash Vector and generates responses via AI SDK + Claude sonnet. The best part: The AI agent uses 𝚝𝚘𝚘𝚕() from AI SDK to retrieve the user's workspace / partner data to provide accurate responses. Finally, if the user needs to speak to a human, we generate a 𝚃𝚒𝚌𝚔𝚎𝚝𝚄𝚙𝚕𝚘𝚊𝚍 UI that lets users upload screenshots + add more context, before creating a ticket on Plain. Cherry on top: The chat interface is embeddable, so we're able to embed it both in our Mintlify docs + support portal 👇 Since we launched this, it's been a gamechanger for scaling support – reducing support workload and creating a much better experience for our users. Highly recommend giving this a try!

We've been obsessing a lot about creating the best possible AI support experience Dub.co. And I think we've found the magic formula. The stack: ✦ Mintlify → docs + help articles ✦ Upstash vector → vector search ✦ AI SDK + Claude sonnet → LLM/retrieval ✦ Plain → support infra Here's how it works: We consolidated our docs + help articles on Mintlify (best-in-class docs platform) so all our support content is now in one place. On every push to main, we sync changes to our Upstash Vector DB. We also built a custom chat interface that retrieves content from Upstash Vector and generates responses via AI SDK + Claude sonnet. The best part: The AI agent uses 𝚝𝚘𝚘𝚕() from AI SDK to retrieve the user's workspace / partner data to provide accurate responses. Finally, if the user needs to speak to a human, we generate a 𝚃𝚒𝚌𝚔𝚎𝚝𝚄𝚙𝚕𝚘𝚊𝚍 UI that lets users upload screenshots + add more context, before creating a ticket on Plain. Cherry on top: The chat interface is embeddable, so we're able to embed it both in our Mintlify docs + support portal 👇 Since we launched this, it's been a gamechanger for scaling support – reducing support workload and creating a much better experience for our users. Highly recommend giving this a try!

Steven Tey

22,284 views • 4 months ago

Also coming up next: s402 SDK To bring on new developers and servers to our platform we will be making a boilerplate/SDK for the ease of integrating s402 payments to their applications. Plus, should be updated and all new bugs are fixed. Feel free to try out an Agent! In our example below we query our BNB Chain memecoin insights tool with our agent. Agents have multi-tool capabilities by the way! Proof it works 👇

Also coming up next: s402 SDK To bring on new developers and servers to our platform we will be making a boilerplate/SDK for the ease of integrating s402 payments to their applications. Plus, should be updated and all new bugs are fixed. Feel free to try out an Agent! In our example below we query our BNB Chain memecoin insights tool with our agent. Agents have multi-tool capabilities by the way! Proof it works 👇

Sora 🔶

14,931 views • 8 months ago

Sometimes the best way to build a UI for an agent is to not build the UI at all. I built an AI agent starter that lives in iMessage. Built on Nitro + Vercel Workflow SDK + evlog - Durable execution - Auto-retry on errors - Full observability

Sometimes the best way to build a UI for an agent is to not build the UI at all. I built an AI agent starter that lives in iMessage. Built on Nitro + Vercel Workflow SDK + evlog - Durable execution - Auto-retry on errors - Full observability

Hugo

11,647 views • 2 months ago

Introducing Paygent.01 The quickest way to spin up an AI Agent that moves real money. Literally just one API key is need and you have your own personal AI Agent that can. - Pay your bills - Send money to that one relative - Go out and ask for money Fully open-source. Built using the AI SDK and Payman.

Introducing Paygent.01 The quickest way to spin up an AI Agent that moves real money. Literally just one API key is need and you have your own personal AI Agent that can. - Pay your bills - Send money to that one relative - Go out and ask for money Fully open-source. Built using the AI SDK and Payman.

tyllen

18,765 views • 1 year ago

🚨 “Tomorrow we’re going to have an announcement - one of the biggest Medical breakthroughs in the history of our Country” “We think we’ve found the answer to Autism & tomorrow Bobby & Health Experts are going to be talking about that” “Parents will be able to help their Child with Autism - their beautiful Child”

🚨 “Tomorrow we’re going to have an announcement - one of the biggest Medical breakthroughs in the history of our Country” “We think we’ve found the answer to Autism & tomorrow Bobby & Health Experts are going to be talking about that” “Parents will be able to help their Child with Autism - their beautiful Child”

Concerned Citizen

5,576,390 views • 9 months ago

We’re excited to announce that we’ve successfully integrated our SDK into the AI Agent Layer | AIFUN platform!🤖 Their ecosystem will now be powered by our technology! 💡The revenue from this integration will be added to our staking pool, details tomorrow. More integrations soon!🤝

We’re excited to announce that we’ve successfully integrated our SDK into the AI Agent Layer | AIFUN platform!🤖 Their ecosystem will now be powered by our technology! 💡The revenue from this integration will be added to our staking pool, details tomorrow. More integrations soon!🤝

AI Voice Agents | AIVA

25,414 views • 1 year ago

Agents & MCP are great... but how can you DEPLOY an agent with MCP tools? Here's the fastest way I've found to build and deploy with the OpenAI Agents SDK in <3 minutes.

Agents & MCP are great... but how can you DEPLOY an agent with MCP tools? Here's the fastest way I've found to build and deploy with the OpenAI Agents SDK in <3 minutes.

matt palmer

64,690 views • 1 year ago

We’ve been working with OpenAI for the past few weeks to test their latest Computer-using Agent model. On our evals, CUA has set a new SOTA. Once integrated with an agent, it can complete long-horizon tasks previously impossible. Try CUA on our playground and Act SDK for free!

We’ve been working with OpenAI for the past few weeks to test their latest Computer-using Agent model. On our evals, CUA has set a new SOTA. Once integrated with an agent, it can complete long-horizon tasks previously impossible. Try CUA on our playground and Act SDK for free!

Scrapybara

89,458 views • 1 year ago

The Claude Code SDK is now the Claude Agent SDK Why? Because we realized the Claude Code agent harness is useful for much more than coding. In fact, we're moving to using it to power most of our own agent loops at Anthropic.

The Claude Code SDK is now the Claude Agent SDK Why? Because we realized the Claude Code agent harness is useful for much more than coding. In fact, we're moving to using it to power most of our own agent loops at Anthropic.

Thariq

206,976 views • 9 months ago

New SDK Integration!🎉 Our technology will be available on the Game Dev Agent platform!🕹 You’ll be able to add sound effects, music, and more to your games!🔊 New Integration = More Rewards for $AIVA Community. Stay tuned for more activities coming from this partnership!

New SDK Integration!🎉 Our technology will be available on the Game Dev Agent platform!🕹 You’ll be able to add sound effects, music, and more to your games!🔊 New Integration = More Rewards for $AIVA Community. Stay tuned for more activities coming from this partnership!

AI Voice Agents | AIVA

16,372 views • 1 year ago

Actions + generative UI is the next major evolution of AI apps. And Vercel just unlocked this for *everybody* with their AI SDK 3.0 update. We’re about to see an explosion of insane AI experiences. This AI stock bot shows where we’re headed.

Actions + generative UI is the next major evolution of AI apps. And Vercel just unlocked this for everybody with their AI SDK 3.0 update. We’re about to see an explosion of insane AI experiences. This AI stock bot shows where we’re headed.

Mckay Wrigley

189,653 views • 2 years ago