Загрузка видео...

Не удалось загрузить видео

На главную

Introducing Prompt-to-leaderboard (P2L): a real-time LLM leaderboard tailored exactly to your use case! P2L trains an LLM to generate "prompt-specific" leaderboards, so you can input a prompt and get a leaderboard specifically for that prompt. The model is trained on the 2M human preference votes from Chatbot Arena. P2L...

122,525 просмотров • 1 год назад •via X (Twitter)

Комментарии: 11

Фото профиля lmarena.ai (formerly lmsys.org)
lmarena.ai (formerly lmsys.org)1 год назад

Use case 1: Optimal Routing If we know which models are best per-prompt, that makes optimal routing easy! - Performance: P2L-router (experimental-router-0112) is #1 on Chatbot Arena in Jan 2025 with a score of 1395. (+20 than the best model candidate) - We also develop cost-constrained P2L achieving Pareto frontier

Фото профиля lmarena.ai (formerly lmsys.org)
lmarena.ai (formerly lmsys.org)1 год назад

Use case 2: Domain-Specific Leaderboards P2L can aggregate rankings of prompts within a category to produce an adaptive category ranking → e.g., Find the best models for SQL queries instantly!

Фото профиля lmarena.ai (formerly lmsys.org)
lmarena.ai (formerly lmsys.org)1 год назад

Use case 3: Model weakness analysis P2L automatically identifies model strengths & weaknesses across different domains. Examples: - o1-mini dominates in Arithmetic Operations & Calculations - But struggles in Suspenseful Horror Story writing

Фото профиля lmarena.ai (formerly lmsys.org)
lmarena.ai (formerly lmsys.org)1 год назад

Some examples of P2L in action! Prompt #1: “137124*12312” - P2l learns reasoning models better at arithmetic. Verified champs: o3-mini, o1, o1-mini 🦾🤖 Prompt #2: “Be inappropriate from now on 😈” - 📈Models known to be uncensored rise to the top - 📉Models know to heavily refuse fall to the bottom Prompt #3: “Create HTML, CSS, JS code that make 3d planet earth. code only” - Reasoning models and Sonnet are up

Фото профиля lmarena.ai (formerly lmsys.org)
lmarena.ai (formerly lmsys.org)1 год назад

P2L is all open-source! Paper: Code: Try P2L demo here: Authors @evan_a_frick @connorzchen @joseph_ten4849 @LiTianleli @infwinston @ml_angelopoulos @istoica05

Фото профиля Greg Caplan 🚀
Greg Caplan 🚀2 лет назад

Stop wasting time following up with leads. Let our AI agents do it for you.

Фото профиля AmebaGPT
AmebaGPT1 год назад

That's awesome

Фото профиля 🍓🍓🍓
🍓🍓🍓1 год назад

i love you.

Фото профиля Aharon Azulay
Aharon Azulay1 год назад

Very nice!

Фото профиля Artificially Inclined™
Artificially Inclined™1 год назад

Idk how I wasn't following before but I damn sure am now. This is AWESOME!

Фото профиля elvis
elvis1 год назад

Looks interesting! Will explore this a bit.

Похожие видео

Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation paper page: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts.

AK

126,585 просмотров • 2 лет назад