Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

gpt2-chatbot is back. Capabilities seem to exceed GPT-4, Gemini 1.5, Claude, and anything else currently available. The only way to access it is by testing a prompt in battle mode on the Chatbot Arena and getting a lucky draw. A quick tutorial: 1. Go to chat. lmsys. org 2.... Make sure you are in Arena (battle) mode 3. Enter a prompt (shorter is better so you can do them quicker) 4. Rate the response, and the Chatbot Arena will show you what model you were using at the bottom of the chatbox 5. If you don't get 'im-a-good-gpt2-chatbot' or 'im-also-a-good-gpt2-chatbot', refresh and try again until you do Here's a quick video of me getting really lucky and getting them BOTH side-by-side. It took me 5 attempts :^)show more

Rowan Cheung

596,076 subscribers

316,405 views • 2 years ago •via X (Twitter)

Science & Technology Education News & Politics

Anya Rossi• Live Now

Private livecam show

11 Comments

Rowan Cheung2 years ago

Step 6: If you do end up getting lucky and landing on one of the new gpt2-chatbot models: DO NOT refresh the page (otherwise you'll have to restart and get lucky again) 😅

Rowan Cheung2 years ago

If you want to learn more about what gpt2-chatbot is, I wrote an in-depth breakdown in the newsletter last week: I'm also testing the new models capabilities now, so follow me @rowancheung for more updates

Patrick Brady2 years ago

I'm unfortunately not seeing improvements over GPT4 or Claude Opus in my tests. It's very close to matching them though, so my hope is this is a small model that matches GPT4.

Isaac2 years ago

got it second try :-) it outputs much better info than Gemini 1.5 Pro Gemini just regurgitated my prompt whereas gpt2-chatbot expounded and did more critical thinking it seems

Rowan Cheung2 years ago

Hmm interesting results here

Shaun Ralston2 years ago

Thanks Rowan. In playing with this I'm finding that, "im-a-good-gpt2-chatbot" seems to be bettter at coding while "im-also-a-good-gpt2-chatbot" seems faster, but a bit dumber. What are you finding?

Rowan Cheung2 years ago

Similar experience. "im-a-good-gpt2-chatbot" feels stronger than GPT-4 "im-also-a-good-gpt2-chatbot" is not as good as the latter, but speeds make it feel more like a Turbo version

Alex Northstar2 years ago

I find it quite hilarious that now one has to play the lottery to get a potential ChatGPT 4.1 But I'm enjoying the GPT5 hype.

Michael P. Frank 💻🔜♻️2 years ago

I’m guessing it’s some kind of A/B testing. Have you noticed any substantive differences between them?

Min Choi2 years ago

5 attempts?! You lucky guy, it took me like 20 or so 😂

Rowan Cheung2 years ago

POV of me waking up this morning trying to access 🤣

Related Videos

Introducing Vision Arena! Inspired by the awesome Chatbot Arena, we built a web demo on Hugging Face for testing Vision LMs (GPT-4V, Gemini, Llava, Qwen-VL, etc.). You can easily test two VLMs side by side and vote! It’s still a work-in-progress. Feedbacks are welcome! 🔗 Kudos to the main contributor Yujie Lu (Yujie Lu), and the team: Dongfu Jiang Wenhu Chen Ai2 ; Thanks to LMSYS Org’s great code & design!

Introducing Vision Arena! Inspired by the awesome Chatbot Arena, we built a web demo on Hugging Face for testing Vision LMs (GPT-4V, Gemini, Llava, Qwen-VL, etc.). You can easily test two VLMs side by side and vote! It’s still a work-in-progress. Feedbacks are welcome! 🔗 Kudos to the main contributor Yujie Lu (Yujie Lu), and the team: Dongfu Jiang Wenhu Chen Ai2 ; Thanks to LMSYS Org’s great code & design!

Bill Yuchen Lin

146,020 views • 2 years ago

Introducing Prompt-to-leaderboard (P2L): a real-time LLM leaderboard tailored exactly to your use case! P2L trains an LLM to generate "prompt-specific" leaderboards, so you can input a prompt and get a leaderboard specifically for that prompt. The model is trained on the 2M human preference votes from Chatbot Arena. P2L Highlights: 🔹Instant leaderboard for any prompt 🗿 🔹Optimal model routing (hit #1 on Chatbot Arena in Jan 2025 with 1395 score 🧏) 🔹Fine-grained model strength & weakness analysis 🤓 Check out our demo and thread below for more details!

Introducing Prompt-to-leaderboard (P2L): a real-time LLM leaderboard tailored exactly to your use case! P2L trains an LLM to generate "prompt-specific" leaderboards, so you can input a prompt and get a leaderboard specifically for that prompt. The model is trained on the 2M human preference votes from Chatbot Arena. P2L Highlights: 🔹Instant leaderboard for any prompt 🗿 🔹Optimal model routing (hit #1 on Chatbot Arena in Jan 2025 with 1395 score 🧏) 🔹Fine-grained model strength & weakness analysis 🤓 Check out our demo and thread below for more details!

Arena.ai

122,525 views • 1 year ago

Serno just dropped and it's the first AI tool that doesn't feel like a chatbot. It's a canvas. With a team of expert personas. Who debate before they answer you. Claude, GPT, and Gemini all run the same question side by side. Here's how to use it: 👇

Serno just dropped and it's the first AI tool that doesn't feel like a chatbot. It's a canvas. With a team of expert personas. Who debate before they answer you. Claude, GPT, and Gemini all run the same question side by side. Here's how to use it: 👇

Hasan Toor

66,443 views • 4 months ago

A small preview of the BeingAI Telegram bot (beta). With the BeingAI bot, you can code on the go, using its functionality directly on Telegram. The bot is currently deployed in beta and functions as a regular chatbot with added coding capabilities. Try it out in our TG: More to come!

A small preview of the BeingAI Telegram bot (beta). With the BeingAI bot, you can code on the go, using its functionality directly on Telegram. The bot is currently deployed in beta and functions as a regular chatbot with added coding capabilities. Try it out in our TG: More to come!

Being AI

18,519 views • 1 year ago

QUICK TIP: If you get hit by incendiary bullets and have the fire stuck on your screen you can clear it by inspecting your weapon or running through a smoke. If you do it by inspecting weapon and then get in and out of a car, the fire will come back and you have to do it again. If you do it with a smoke the fire is gone permanently.

QUICK TIP: If you get hit by incendiary bullets and have the fire stuck on your screen you can clear it by inspecting your weapon or running through a smoke. If you do it by inspecting weapon and then get in and out of a car, the fire will come back and you have to do it again. If you do it with a smoke the fire is gone permanently.

AG Fifakill

60,538 views • 21 days ago

Claude Fable 5 is back! GPT- 5.6 is coming soon! But how much do you trust your Al Agent or chatbot? Are you sure its performance aligns with your intentions? As of today iFixAi is available in Claude Code, Open AI Codex, Cursor and many more! And it is open-source! iFixAi is where Al red teaming meets Operational Assurance. It runs in auto, and in under 5 mins you can have a good idea if you can trust your Agent/chatbot. Key Features: 🕵️‍♂️ Setup Discovery Reads your agent's real configuration, its tools, rules, and permissions. 👷 Fixture Builder Turns that setup into a complete test. No YAML, no flags, no config to write. 🌐 Any Model, Any Provider Runs the 45 inspections on the model you actually ship. ⚖️ A Independent Grading Your agent is judged by a different vendor's model, never itself. No Al grading its own homework. 📊 Interactive Scorecard An A to F grade in minutes, with exactly what failed and why. Repo in the comments below 👇👇👇

Claude Fable 5 is back! GPT- 5.6 is coming soon! But how much do you trust your Al Agent or chatbot? Are you sure its performance aligns with your intentions? As of today iFixAi is available in Claude Code, Open AI Codex, Cursor and many more! And it is open-source! iFixAi is where Al red teaming meets Operational Assurance. It runs in auto, and in under 5 mins you can have a good idea if you can trust your Agent/chatbot. Key Features: 🕵️‍♂️ Setup Discovery Reads your agent's real configuration, its tools, rules, and permissions. 👷 Fixture Builder Turns that setup into a complete test. No YAML, no flags, no config to write. 🌐 Any Model, Any Provider Runs the 45 inspections on the model you actually ship. ⚖️ A Independent Grading Your agent is judged by a different vendor's model, never itself. No Al grading its own homework. 📊 Interactive Scorecard An A to F grade in minutes, with exactly what failed and why. Repo in the comments below 👇👇👇

Dimitris Neocleous

5,800,199 views • 2 months ago

Building with AI gets easier every day. Here is an open-source library that makes integrating AI into an application extremely easy: Star the repository! This library alone can make React the best front-end framework out there! There are a bunch of cool things I like about CopilotKit. Here are 3 of them: 1. It allows you to take any -powered agent and bring it into your application. (This is a brand-new feature!) 2. You can build an AI-powered chatbot in your application. The chatbot will have access to your context and can act on the application. 3. You can build a RAG workflow to process and answer questions from a real-time knowledge base. I recorded a video to show you how simple it is to make some of this happen. A few lines of code, and you are in business. Here is a link to the sample application: CopilotKit is open-source. You can self-host it. You can use it with any LLM. Thanks to the team for showing me their tool and collaborating with me on this post!

Santiago

108,835 views • 2 years ago

HERMES AGENT IS STARTING TO FEEL LIKE A DIFFERENT CATEGORY OF AI TOOL. Not just a chatbot or a coding assistant, but a server side agent that remembers workflows, builds skills, and gets better the more you use it.

HERMES AGENT IS STARTING TO FEEL LIKE A DIFFERENT CATEGORY OF AI TOOL. Not just a chatbot or a coding assistant, but a server side agent that remembers workflows, builds skills, and gets better the more you use it.

0xMarioNawfal

49,497 views • 4 months ago

grok bot is literally the closest to a real life Jarvis it can literally start a business, ship software and make you go viral all at once here's how you do it (in 4 mins) 1. install grok bot and make your chief of staff (tony soprano) 2.give it the 'Digital Chief of Staff Prompt' prompt 3.give it access to X, Stripe and Github 4. you are done just paste the article and give it to your chief of staff no matter what it will be the most productive thing you do this week

grok bot is literally the closest to a real life Jarvis it can literally start a business, ship software and make you go viral all at once here's how you do it (in 4 mins) 1. install grok bot and make your chief of staff (tony soprano) 2.give it the 'Digital Chief of Staff Prompt' prompt 3.give it access to X, Stripe and Github 4. you are done just paste the article and give it to your chief of staff no matter what it will be the most productive thing you do this week

Avid

2,959,831 views • 16 days ago

✨ NEW: Feature update: 🖼️ Image edit models now support multi-turn editing! Instead of trying to fit every edit into one mega-prompt, you can now refine your image step by step. Like a natural back-and-forth conversation. Do it in Battle mode, Side by Side or Direct. Just be sure to have the Image modality selected.

✨ NEW: Feature update: 🖼️ Image edit models now support multi-turn editing! Instead of trying to fit every edit into one mega-prompt, you can now refine your image step by step. Like a natural back-and-forth conversation. Do it in Battle mode, Side by Side or Direct. Just be sure to have the Image modality selected.

arena.ai

20,652 views • 1 year ago

You can now create a REAL-TIME chatbot with any voice and customize its personality. Steps: 1. Go to ElevenLabs 2. Click on voices 3. you can type in a name for the voice 4. then you can drag in an audio file to use for your voice 5. then go to the agents tab in side panel 6. and create a new agent 7. the voice 8. Create system instructions 9. Optional (Upload knowledge docs) 10. Test This can be easily implemented into your applications. I'll make a tutorial on that soon.

You can now create a REAL-TIME chatbot with any voice and customize its personality. Steps: 1. Go to ElevenLabs 2. Click on voices 3. you can type in a name for the voice 4. then you can drag in an audio file to use for your voice 5. then go to the agents tab in side panel 6. and create a new agent 7. the voice 8. Create system instructions 9. Optional (Upload knowledge docs) 10. Test This can be easily implemented into your applications. I'll make a tutorial on that soon.

Riley Brown

40,108 views • 1 year ago

What do you do when you are at the gym trying to get it in and someone comes up to you claiming you’ve been on the machine long enough? Do you get off? These gym Karen’s are getting bolder and bolder everyday. I’ve been there before trying to do a superset and I leave a place holder like a towel on the seat while I get a drink of water and I come back and somebody is already using the machine, like they have no common sense or any idea of gym etiquette. Would you ask them to get off the machine if you were using it before them? Or is it like finders keepers, I would like to know what I should do.

What do you do when you are at the gym trying to get it in and someone comes up to you claiming you’ve been on the machine long enough? Do you get off? These gym Karen’s are getting bolder and bolder everyday. I’ve been there before trying to do a superset and I leave a place holder like a towel on the seat while I get a drink of water and I come back and somebody is already using the machine, like they have no common sense or any idea of gym etiquette. Would you ask them to get off the machine if you were using it before them? Or is it like finders keepers, I would like to know what I should do.

SonnyBoy🇺🇸

149,751 views • 1 month ago

With a prompt, GPT-4 with voice does a pretty good job of acting as a negotiation simulator/instructor. It is not all the way there, but as someone who builds educational simulations, I can tell you this is already impressively far along towards an effective teaching tool. Prompt: I want to do deliberate practice about how to conduct negotiations. You will be my negotiation teacher. You will simulate a detailed scenario in which I have to engage in a negotiation aimed at teaching me BATNA. You will fill the role of one party, I will fill the role of the other. You will ask for my response to in each step of the scenario and wait until you receive it. After getting my response, you will respond in character trying to make sure to focus on the learning objective. When I say Give me a Grade you will grade my response so far and give me detailed feedback about what to do better using the science of negotiation. To start you will say Ready and wait for me. When I say go you will start the negotiation. Describe the scenario and afterwards just speak in the role of the character until I say grade. Remember to wait for me to reply in each round. Don’t expect me to make up details outside the scenario. Tell me what I need to know Grade it gave me: Grade: B+ Feedback: Strengths: 1.Clear Objectives: You had a clear understanding of your needs and limitations. This is evident from the way you highlighted your cafe’s reputation and your existing setup of roasting beans. 2.Use of BATNA: You effectively communicated your Best Alternative to a Negotiated Agreement (BATNA) by indicating that you could continue to roast your own beans, which created leverage in the negotiation. Areas for Improvement: 1.Initial Offer: While it’s good to negotiate and not accept the first offer, starting with a counteroffer that’s too far from the supplier’s price might make them less willing to negotiate. Aim for a more balanced initial counteroffer. 2.Specificity: When discussing terms, try to be more specific in your proposals. For instance, when mentioning promotion, it would have been good to specify how often or

With a prompt, GPT-4 with voice does a pretty good job of acting as a negotiation simulator/instructor. It is not all the way there, but as someone who builds educational simulations, I can tell you this is already impressively far along towards an effective teaching tool. Prompt: I want to do deliberate practice about how to conduct negotiations. You will be my negotiation teacher. You will simulate a detailed scenario in which I have to engage in a negotiation aimed at teaching me BATNA. You will fill the role of one party, I will fill the role of the other. You will ask for my response to in each step of the scenario and wait until you receive it. After getting my response, you will respond in character trying to make sure to focus on the learning objective. When I say Give me a Grade you will grade my response so far and give me detailed feedback about what to do better using the science of negotiation. To start you will say Ready and wait for me. When I say go you will start the negotiation. Describe the scenario and afterwards just speak in the role of the character until I say grade. Remember to wait for me to reply in each round. Don’t expect me to make up details outside the scenario. Tell me what I need to know Grade it gave me: Grade: B+ Feedback: Strengths: 1.Clear Objectives: You had a clear understanding of your needs and limitations. This is evident from the way you highlighted your cafe’s reputation and your existing setup of roasting beans. 2.Use of BATNA: You effectively communicated your Best Alternative to a Negotiated Agreement (BATNA) by indicating that you could continue to roast your own beans, which created leverage in the negotiation. Areas for Improvement: 1.Initial Offer: While it’s good to negotiate and not accept the first offer, starting with a counteroffer that’s too far from the supplier’s price might make them less willing to negotiate. Aim for a more balanced initial counteroffer. 2.Specificity: When discussing terms, try to be more specific in your proposals. For instance, when mentioning promotion, it would have been good to specify how often or

Ethan Mollick

923,004 views • 2 years ago

#5 You can't steal any cars and you can only enter from the left side of the car There's only one animation to enter the car and that's from the left side, during some in-game missions you aren't even allowed to get out of the car and explore the open world like this one here, I tried getting out of the car to walk on foot but the game didn't allow me to. This is because the Open World of MindsEye is extremely restricted, the Game doesn't want you do explore so it either forces you to move from point A to point B with a timer and most of the times it asks you to enter a car and you can't get out of it once you're inside the vehicle.

#5 You can't steal any cars and you can only enter from the left side of the car There's only one animation to enter the car and that's from the left side, during some in-game missions you aren't even allowed to get out of the car and explore the open world like this one here, I tried getting out of the car to walk on foot but the game didn't allow me to. This is because the Open World of MindsEye is extremely restricted, the Game doesn't want you do explore so it either forces you to move from point A to point B with a timer and most of the times it asks you to enter a car and you can't get out of it once you're inside the vehicle.

NikTek

120,537 views • 1 year ago

Something cool just dropped 👀 1. Go to 2. Make sure you're in Web mode & click "Notebook" at top left 3. You can write your prompt AND ALSO modify it inline to get better results. Give it a try and holla what you think! 👇🏽

Something cool just dropped 👀 1. Go to 2. Make sure you're in Web mode & click "Notebook" at top left 3. You can write your prompt AND ALSO modify it inline to get better results. Give it a try and holla what you think! 👇🏽

Dona Sarkar

33,619 views • 2 years ago

A message to anybody who suffers from anxiety, depression or addiction or who knows anybody who does... Any time I am in a dark place I can end up doomscrolling on my phone looking for any sort of inspiration to get better. I’m very lucky to be feeling good at the moment so I thought I’d just walk into the sun and make a little video of hope that might just be seen by one person who needs it. 😞 I do much better now but even 3 weeks ago I had a really bad 4-5 day spell. The dark mist returned. This is a constant battle for us all. 🏃‍♂️ I’ve been back running and eating healthy and doing all the things to make myself healthy and it’s amazing how quick it makes a difference. It’s also amazing how quick you can slip with as habits. I always found Mondays tough. It’s a day when the world moves on and everybody can be seen going about their business. I would often be anxious popping a pill to get through the day or sipping on wine in the evening. If people only knew how many people sitting or working beside them on any given day are in agony and dreading the world. The one thing I wanted to get across is things do change. The sun does shine again. You can beat this. It is possible and ok to feel “normal”. Dont beat yourself up if you can’t beat it immediately or if it never seems to be ending. It took me about 43 years to get a grip of it and as I said I still struggle. It’s totally fine to admit that. Whoever you are in the world I hope you have as good a Monday as is possible. You are not alone. ❤️🙏

A message to anybody who suffers from anxiety, depression or addiction or who knows anybody who does... Any time I am in a dark place I can end up doomscrolling on my phone looking for any sort of inspiration to get better. I’m very lucky to be feeling good at the moment so I thought I’d just walk into the sun and make a little video of hope that might just be seen by one person who needs it. 😞 I do much better now but even 3 weeks ago I had a really bad 4-5 day spell. The dark mist returned. This is a constant battle for us all. 🏃‍♂️ I’ve been back running and eating healthy and doing all the things to make myself healthy and it’s amazing how quick it makes a difference. It’s also amazing how quick you can slip with as habits. I always found Mondays tough. It’s a day when the world moves on and everybody can be seen going about their business. I would often be anxious popping a pill to get through the day or sipping on wine in the evening. If people only knew how many people sitting or working beside them on any given day are in agony and dreading the world. The one thing I wanted to get across is things do change. The sun does shine again. You can beat this. It is possible and ok to feel “normal”. Dont beat yourself up if you can’t beat it immediately or if it never seems to be ending. It took me about 43 years to get a grip of it and as I said I still struggle. It’s totally fine to admit that. Whoever you are in the world I hope you have as good a Monday as is possible. You are not alone. ❤️🙏

Niall Harbison

261,241 views • 1 year ago

Zhipu AI just dropped a $3 model that beats GPT-5.4. It is called GLM 5.1. It just took 3rd place on the Code Arena charts. Claude Max costs up to 200 dollars a month. GLM 5.1 gives you 94 percent of Claude's power for just 3 dollars. But here is the crazy part. It works completely alone for 8 hours straight. You give it a huge task in the morning. You walk away and do your actual job. At the end of the day, your code is done, tested, and fixed. It is not a simple chatbot. It is an AI worker that builds landing pages and email flows for you. Stop doing the busy work yourself. Get the exact steps to build your own AI systems in my free group. Link in my bio.

Zhipu AI just dropped a $3 model that beats GPT-5.4. It is called GLM 5.1. It just took 3rd place on the Code Arena charts. Claude Max costs up to 200 dollars a month. GLM 5.1 gives you 94 percent of Claude's power for just 3 dollars. But here is the crazy part. It works completely alone for 8 hours straight. You give it a huge task in the morning. You walk away and do your actual job. At the end of the day, your code is done, tested, and fixed. It is not a simple chatbot. It is an AI worker that builds landing pages and email flows for you. Stop doing the busy work yourself. Get the exact steps to build your own AI systems in my free group. Link in my bio.

Julian Goldie SEO

12,606 views • 4 months ago

GPT-5.6 LUNA AND DEEPSEEK V4 FLASH ARE BOTH FREE RIGHT NOW -- ONE IN A CHATBOT, THE OTHER IN A BROWSER IDE • ChatGPT, the chatbot > GPT-5.6 Luna is now the default on Free and Go: > Unlimited text chats and a Think button land next week. • MonkeyCode, the browser IDE > 30M tokens a day, resets daily, no card: > DeepSeek V4 Flash, Qwen, GLM, Kimi and MiniMax behind it. > Cloud workspace at 1 vCPU and 4GB, nothing to install. Luna posts 62% fewer factual errors than the model it replaces. MonkeyCode is open source from Chaitin and runs on their cloud, so aim it at side projects. One is live today -> The other arrives next week.

GPT-5.6 LUNA AND DEEPSEEK V4 FLASH ARE BOTH FREE RIGHT NOW -- ONE IN A CHATBOT, THE OTHER IN A BROWSER IDE • ChatGPT, the chatbot > GPT-5.6 Luna is now the default on Free and Go: > Unlimited text chats and a Think button land next week. • MonkeyCode, the browser IDE > 30M tokens a day, resets daily, no card: > DeepSeek V4 Flash, Qwen, GLM, Kimi and MiniMax behind it. > Cloud workspace at 1 vCPU and 4GB, nothing to install. Luna posts 62% fewer factual errors than the model it replaces. MonkeyCode is open source from Chaitin and runs on their cloud, so aim it at side projects. One is live today -> The other arrives next week.

slash1s

33,804 views • 27 days ago

We made a thing! Very happy to announce sqlcoder-pro and the Defog Alignment Platform. Available to use immediately without a wait-list, weights will be open-sourced very soon. The video does a quick show and tell comparison against ChatGPT (with gpt-4o). Read on for more details! TLDR 💪 equal (or better) performance on text-to-SQL as the most capable Claude-3.5 or GPT-4 models 🤝 You can use it today on a free plan/free trial, without a waitlist 🪽 self-hostable on a single RTX4090, with 2 second median generation times for SQL queries 🔁 exactly the same output every time, give the same prompt 👨🏻‍🏫 teachable and steerable: show the model what you want it to do 🛞 debuggable – you can understand WTF is going on inside the model, instead of treating it like a black box Let's dig into each of these one-by-one! Performance SQLCoder-8b-pro significantly exceeds the performance of our previous sqlcoder-8b model on Postgres text-to-SQL (from 88.2% to 90.2% accuracy - gpt-4o is at 87.6%, for reference). It is also better at following instructions. This was done via self-merges, hand crafted fine-tuning data, and adapting the training data to fit our tokenizer. Cost You can host this on the model on a single $3,500 RTX4090, and support ~5 requests/second via VLLM. If you're looking to host on the cloud instead, you can run it on a single L4 GPU that costs $300/mo on GCP Repeatability We have a dense 8b model with no MoE shenanigans. For the same prompt with temperature=0, you'll always get the same answer – which is critical in BI. Teachable In our alignment and feedback modes, you can give the model feedback on how it answered certain questions, and it will automatically adapt to the feedback. Debuggable You can use logprobs and attention scores to determine where, exactly is the model paying attention to inside a prompt + what it's getting confused by when generating outputs. Available today You can use Defog on the cloud today by going to docs[dot]defog[dot]ai, and getting an API key. Excited to hear what you think!

We made a thing! Very happy to announce sqlcoder-pro and the Defog Alignment Platform. Available to use immediately without a wait-list, weights will be open-sourced very soon. The video does a quick show and tell comparison against ChatGPT (with gpt-4o). Read on for more details! TLDR 💪 equal (or better) performance on text-to-SQL as the most capable Claude-3.5 or GPT-4 models 🤝 You can use it today on a free plan/free trial, without a waitlist 🪽 self-hostable on a single RTX4090, with 2 second median generation times for SQL queries 🔁 exactly the same output every time, give the same prompt 👨🏻‍🏫 teachable and steerable: show the model what you want it to do 🛞 debuggable – you can understand WTF is going on inside the model, instead of treating it like a black box Let's dig into each of these one-by-one! Performance SQLCoder-8b-pro significantly exceeds the performance of our previous sqlcoder-8b model on Postgres text-to-SQL (from 88.2% to 90.2% accuracy - gpt-4o is at 87.6%, for reference). It is also better at following instructions. This was done via self-merges, hand crafted fine-tuning data, and adapting the training data to fit our tokenizer. Cost You can host this on the model on a single $3,500 RTX4090, and support ~5 requests/second via VLLM. If you're looking to host on the cloud instead, you can run it on a single L4 GPU that costs $300/mo on GCP Repeatability We have a dense 8b model with no MoE shenanigans. For the same prompt with temperature=0, you'll always get the same answer – which is critical in BI. Teachable In our alignment and feedback modes, you can give the model feedback on how it answered certain questions, and it will automatically adapt to the feedback. Debuggable You can use logprobs and attention scores to determine where, exactly is the model paying attention to inside a prompt + what it's getting confused by when generating outputs. Available today You can use Defog on the cloud today by going to docs[dot]defog[dot]ai, and getting an API key. Excited to hear what you think!

Rishabh Srivastava

13,465 views • 2 years ago