Loading video...

Video Failed to Load

Go Home

Introducing Prompt-to-leaderboard (P2L): a real-time LLM leaderboard tailored exactly to your use case! P2L trains an LLM to generate "prompt-specific" leaderboards, so you can input a prompt and get a leaderboard specifically for that prompt. The model is trained on the 2M human preference votes from Chatbot Arena. P2L...

122,525 views • 1 year ago •via X (Twitter)

11 Comments

lmarena.ai (formerly lmsys.org)'s profile picture
lmarena.ai (formerly lmsys.org)1 year ago

Use case 1: Optimal Routing If we know which models are best per-prompt, that makes optimal routing easy! - Performance: P2L-router (experimental-router-0112) is #1 on Chatbot Arena in Jan 2025 with a score of 1395. (+20 than the best model candidate) - We also develop cost-constrained P2L achieving Pareto frontier

lmarena.ai (formerly lmsys.org)'s profile picture
lmarena.ai (formerly lmsys.org)1 year ago

Use case 2: Domain-Specific Leaderboards P2L can aggregate rankings of prompts within a category to produce an adaptive category ranking → e.g., Find the best models for SQL queries instantly!

lmarena.ai (formerly lmsys.org)'s profile picture
lmarena.ai (formerly lmsys.org)1 year ago

Use case 3: Model weakness analysis P2L automatically identifies model strengths & weaknesses across different domains. Examples: - o1-mini dominates in Arithmetic Operations & Calculations - But struggles in Suspenseful Horror Story writing

lmarena.ai (formerly lmsys.org)'s profile picture
lmarena.ai (formerly lmsys.org)1 year ago

Some examples of P2L in action! Prompt #1: “137124*12312” - P2l learns reasoning models better at arithmetic. Verified champs: o3-mini, o1, o1-mini 🦾🤖 Prompt #2: “Be inappropriate from now on 😈” - 📈Models known to be uncensored rise to the top - 📉Models know to heavily refuse fall to the bottom Prompt #3: “Create HTML, CSS, JS code that make 3d planet earth. code only” - Reasoning models and Sonnet are up

lmarena.ai (formerly lmsys.org)'s profile picture
lmarena.ai (formerly lmsys.org)1 year ago

P2L is all open-source! Paper: Code: Try P2L demo here: Authors @evan_a_frick @connorzchen @joseph_ten4849 @LiTianleli @infwinston @ml_angelopoulos @istoica05

Greg Caplan 🚀's profile picture
Greg Caplan 🚀2 years ago

Stop wasting time following up with leads. Let our AI agents do it for you.

AmebaGPT's profile picture
AmebaGPT1 year ago

That's awesome

🍓🍓🍓's profile picture
🍓🍓🍓1 year ago

i love you.

Aharon Azulay's profile picture
Aharon Azulay1 year ago

Very nice!

Artificially Inclined™'s profile picture
Artificially Inclined™1 year ago

Idk how I wasn't following before but I damn sure am now. This is AWESOME!

elvis's profile picture
elvis1 year ago

Looks interesting! Will explore this a bit.

Related Videos

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

AK

126,585 views • 2 years ago