正在加载视频...

视频加载失败

Introducing Prompt-to-leaderboard (P2L): a real-time LLM leaderboard tailored exactly to your use case! P2L trains an LLM to generate "prompt-specific" leaderboards, so you can input a prompt and get a leaderboard specifically for that prompt. The model is trained on the 2M human preference votes from Chatbot Arena. P2L...

122,525 次观看 • 1 年前 •via X (Twitter)

11 条评论

lmarena.ai (formerly lmsys.org) 的头像
lmarena.ai (formerly lmsys.org)1 年前

Use case 1: Optimal Routing If we know which models are best per-prompt, that makes optimal routing easy! - Performance: P2L-router (experimental-router-0112) is #1 on Chatbot Arena in Jan 2025 with a score of 1395. (+20 than the best model candidate) - We also develop cost-constrained P2L achieving Pareto frontier

lmarena.ai (formerly lmsys.org) 的头像
lmarena.ai (formerly lmsys.org)1 年前

Use case 2: Domain-Specific Leaderboards P2L can aggregate rankings of prompts within a category to produce an adaptive category ranking → e.g., Find the best models for SQL queries instantly!

lmarena.ai (formerly lmsys.org) 的头像
lmarena.ai (formerly lmsys.org)1 年前

Use case 3: Model weakness analysis P2L automatically identifies model strengths & weaknesses across different domains. Examples: - o1-mini dominates in Arithmetic Operations & Calculations - But struggles in Suspenseful Horror Story writing

lmarena.ai (formerly lmsys.org) 的头像
lmarena.ai (formerly lmsys.org)1 年前

Some examples of P2L in action! Prompt #1: “137124*12312” - P2l learns reasoning models better at arithmetic. Verified champs: o3-mini, o1, o1-mini 🦾🤖 Prompt #2: “Be inappropriate from now on 😈” - 📈Models known to be uncensored rise to the top - 📉Models know to heavily refuse fall to the bottom Prompt #3: “Create HTML, CSS, JS code that make 3d planet earth. code only” - Reasoning models and Sonnet are up

lmarena.ai (formerly lmsys.org) 的头像
lmarena.ai (formerly lmsys.org)1 年前

P2L is all open-source! Paper: Code: Try P2L demo here: Authors @evan_a_frick @connorzchen @joseph_ten4849 @LiTianleli @infwinston @ml_angelopoulos @istoica05

Greg Caplan 🚀 的头像
Greg Caplan 🚀2 年前

Stop wasting time following up with leads. Let our AI agents do it for you.

AmebaGPT 的头像
AmebaGPT1 年前

That's awesome

🍓🍓🍓 的头像
🍓🍓🍓1 年前

i love you.

Aharon Azulay 的头像
Aharon Azulay1 年前

Very nice!

Artificially Inclined™ 的头像
Artificially Inclined™1 年前

Idk how I wasn't following before but I damn sure am now. This is AWESOME!

elvis 的头像
elvis1 年前

Looks interesting! Will explore this a bit.

相关视频

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

AK

126,585 次观看 • 2 年前