DPO Debate: Is RL needed for RLHF?
@natolambert102K followers
100,049 views
11 Comments
Comments from the original post on X (Twitter)

@natolambert
here we go again @abacaj @ericmitchellai @srush_nlp @_lewtun @hamishivi @rajammanabrolu @teortaxesTex @yacineMTB @erhartford @Teknium1 @ethayarajh
9 likes

@AllanZhou17
If the derivation of Eq 4 is a bit esoteric, there's always the more direct (but tedious) approach: form the Lagrangian L(π)=E[r(y)]-βKL(π||π_ref)+λ(∑π(y)-1) and set dL/dπ(y)=0, solve.
7 likes

@natolambert
Can you post that derivation too?

@cheng_pengyu
Nice summary!BTW, please also take a glance at our APO (Adversarial Preference Optimization) at which is an adversarial alignment method and compatible to either DPO/IPO/PPO😃
7 likes

@_RanW_
“Language tasks are trees” I finally found it! Second guessing: what about goal-oriented dialogues, having the ability to backspace, etc.
6 likes

@natolambert
Goal oriented as you describe it prolly works, but I'm not a theory person! Good point!
1 like

@Learnius
@ylecun @natolambert have you read the IPO paper carefully? We had chats with main authors of DPO and they are aware of limitations of DPO, i.e., Bradly Theory assumption and overfitting. DPO is a pioneering work in this field and in IPO we try to improve it by alleviating limitations.
2 likes

@natolambert
@ylecun I read it a good bit, but definitely not all. Two things stuck out that made me turn away from complete read: * It came off as pretty aggressive towards DPO, which is a different style of paper to read * It still needs a lot of experiments I'm happy to keep learning more

@snowflect
@ylecun Don’t forget the #GazaGenocide and #IsraelCriminalWar and ongoing atrocities
1 like

@Learnius
I am not sure expressing quite negative comments about a work (based on softballs like "there is no experiments, which is not true") either help the advancement of the field or help people gain better understanding of the topic.
1 like

@natolambert
you're right, followed up via email.
1 like
