Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Introducing Prompt-to-leaderboard (P2L): a real-time LLM leaderboard tailored exactly to your use case! P2L trains an LLM to generate "prompt-specific" leaderboards, so you can input a prompt and get a leaderboard specifically for that prompt. The model is trained on the 2M human preference votes from Chatbot Arena. P2L...

122,525 Aufrufe • vor 1 Jahr •via X (Twitter)

11 Kommentare

Profilbild von lmarena.ai (formerly lmsys.org)
lmarena.ai (formerly lmsys.org)vor 1 Jahr

Use case 1: Optimal Routing If we know which models are best per-prompt, that makes optimal routing easy! - Performance: P2L-router (experimental-router-0112) is #1 on Chatbot Arena in Jan 2025 with a score of 1395. (+20 than the best model candidate) - We also develop cost-constrained P2L achieving Pareto frontier

Profilbild von lmarena.ai (formerly lmsys.org)
lmarena.ai (formerly lmsys.org)vor 1 Jahr

Use case 2: Domain-Specific Leaderboards P2L can aggregate rankings of prompts within a category to produce an adaptive category ranking → e.g., Find the best models for SQL queries instantly!

Profilbild von lmarena.ai (formerly lmsys.org)
lmarena.ai (formerly lmsys.org)vor 1 Jahr

Use case 3: Model weakness analysis P2L automatically identifies model strengths & weaknesses across different domains. Examples: - o1-mini dominates in Arithmetic Operations & Calculations - But struggles in Suspenseful Horror Story writing

Profilbild von lmarena.ai (formerly lmsys.org)
lmarena.ai (formerly lmsys.org)vor 1 Jahr

Some examples of P2L in action! Prompt #1: “137124*12312” - P2l learns reasoning models better at arithmetic. Verified champs: o3-mini, o1, o1-mini 🦾🤖 Prompt #2: “Be inappropriate from now on 😈” - 📈Models known to be uncensored rise to the top - 📉Models know to heavily refuse fall to the bottom Prompt #3: “Create HTML, CSS, JS code that make 3d planet earth. code only” - Reasoning models and Sonnet are up

Profilbild von lmarena.ai (formerly lmsys.org)
lmarena.ai (formerly lmsys.org)vor 1 Jahr

P2L is all open-source! Paper: Code: Try P2L demo here: Authors @evan_a_frick @connorzchen @joseph_ten4849 @LiTianleli @infwinston @ml_angelopoulos @istoica05

Profilbild von Greg Caplan 🚀
Greg Caplan 🚀vor 2 Jahren

Stop wasting time following up with leads. Let our AI agents do it for you.

Profilbild von AmebaGPT
AmebaGPTvor 1 Jahr

That's awesome

Profilbild von 🍓🍓🍓
🍓🍓🍓vor 1 Jahr

i love you.

Profilbild von Aharon Azulay
Aharon Azulayvor 1 Jahr

Very nice!

Profilbild von Artificially Inclined™
Artificially Inclined™vor 1 Jahr

Idk how I wasn't following before but I damn sure am now. This is AWESOME!

Profilbild von elvis
elvisvor 1 Jahr

Looks interesting! Will explore this a bit.

Ähnliche Videos

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

AK

126,585 Aufrufe • vor 2 Jahren