Skip navigation
DPO Debate: Is RL needed for RLHF?
0:00 / 26:54

DPO Debate: Is RL needed for RLHF?

Nathan Lambert

@natolambert102K followers

Visit on X
429 likesDownloadOriginal post

100,049 views

DPO Debate: Is RL needed for RLHF? All things as we cannot settle if DPO or RL is better. At least it is a good exercise. 1. Derivations in the DPO paper. Hint, the authors are good at math 2. cDPO, IPO, and related equations 3. Speculation on potential oddities of DPO vs RL 4. Reminders on the state of open RLHF tldr: we have more limitations with data and tooling and evaluation than optimizer choice Slides: Recent blog post of mine on DPO (more next Wed.): DPO Paper: On youtube:

11 Comments

Comments from the original post on X (Twitter)

  • Nathan Lambert's profile picture

    @natolambert

    here we go again @abacaj @ericmitchellai @srush_nlp @_lewtun @hamishivi @rajammanabrolu @teortaxesTex @yacineMTB @erhartford @Teknium1 @ethayarajh

    9 likes

  • Allan Zhou's profile picture

    @AllanZhou17

    If the derivation of Eq 4 is a bit esoteric, there's always the more direct (but tedious) approach: form the Lagrangian L(π)=E[r(y)]-βKL(π||π_ref)+λ(∑π(y)-1) and set dL/dπ(y)=0, solve.

    7 likes

  • Nathan Lambert's profile picture

    @natolambert

    Can you post that derivation too?

  • Pengyu Cheng's profile picture

    @cheng_pengyu

    Nice summary!BTW, please also take a glance at our APO (Adversarial Preference Optimization) at which is an adversarial alignment method and compatible to either DPO/IPO/PPO😃

    7 likes

  • RanW's profile picture

    @_RanW_

    “Language tasks are trees” I finally found it! Second guessing: what about goal-oriented dialogues, having the ability to backspace, etc.

    6 likes

  • Nathan Lambert's profile picture

    @natolambert

    Goal oriented as you describe it prolly works, but I'm not a theory person! Good point!

    1 like

  • Mohammad Azar's profile picture

    @Learnius

    @ylecun @natolambert have you read the IPO paper carefully? We had chats with main authors of DPO and they are aware of limitations of DPO, i.e., Bradly Theory assumption and overfitting. DPO is a pioneering work in this field and in IPO we try to improve it by alleviating limitations.

    2 likes

  • Nathan Lambert's profile picture

    @natolambert

    @ylecun I read it a good bit, but definitely not all. Two things stuck out that made me turn away from complete read: * It came off as pretty aggressive towards DPO, which is a different style of paper to read * It still needs a lot of experiments I'm happy to keep learning more

  • Snowflect's profile picture

    @snowflect

    @ylecun Don’t forget the #GazaGenocide and #IsraelCriminalWar and ongoing atrocities

    1 like

  • Mohammad Azar's profile picture

    @Learnius

    I am not sure expressing quite negative comments about a work (based on softballs like "there is no experiments, which is not true") either help the advancement of the field or help people gain better understanding of the topic.

    1 like

  • Nathan Lambert's profile picture

    @natolambert

    you're right, followed up via email.

    1 like