正在加载视频...

视频加载失败

We present Tropical Reinforcement Learning. Standard RL for LLMs cannot learn compositions effectively, for a fundamental reason: its algebra is not optimal. It sums the probabilities of every successful answer, measuring how often a model succeeds, not which solution succeeds.

43,127 次观看 • 1 个月前 •via X (Twitter)

22 条评论

Arip 的头像
Arip1 个月前

Our fix: change the algebra. Instead of summing over solutions, take the max → the tropical (max-plus) semiring. Each state's value is now the log-prob of its best verified solution, with the path itself. Good prefixes and suffixes from different rollouts can be joined.

Arip 的头像
Arip1 个月前

Results: TROPIC beats the strongest on-policy baselines by up to 16 percentage points across four benchmarks. Big picture: changing the algebra of RL, not just its estimators, can make LLMs much better at compositional reasoning.

Arip 的头像
Arip1 个月前

Paper: Code:

Swaroop Nath 的头像
Swaroop Nath1 个月前

This is insane clickbait 🙂 I thought SSI released something

Arip 的头像
Arip1 个月前

Sorry, I didn’t even think someone would see it that way.

Swaroop Nath 的头像
Swaroop Nath1 个月前

It's no problem! Nice work, tho!

Joshua V. Dillon 的头像
Joshua V. Dillon1 个月前

Very similar to our agentic discovery on sudoku/arc. Although our focus was max likelihood not RL, the idea is the same: backprop through the winning head but transfer state to losers too so rich don't get richer.

Ryan McCormick 的头像
Ryan McCormick1 个月前

This is exactly right. Well trained means to represent the underlying algebra precisely.

Saksham Jain 的头像
Saksham Jain1 个月前

if this holds, a lot of the pass@k shrinkage people pin on rl is really just the sum being the wrong thing to optimize

Arip 的头像
Arip1 个月前

More results on pass@k soon.

Saksham Jain 的头像
Saksham Jain1 个月前

looking forward to it. does the gap keep widening as k grows or does it level off past some point?

Catch Vartan 的头像
Catch Vartan1 个月前

Fun fact: the tropical semiring was named such because it came from Brazil and the French mathematicians who saw just thought it was wild they were getting mathematics out of Brazil.

Arip 的头像
Arip1 个月前

Yeah, max-plus is used more often, but I prefer tropical

Henrique Dalposso 的头像
Henrique Dalposso1 个月前

It makes the LLM go bananas

Arip 的头像
Arip1 个月前

😂😂

Greer 的头像
Greer1 个月前

Finally, congrats on the release! 🎉

Carter Leffen 的头像
Carter Leffen1 个月前

@tairov you had me at Ilya and had me through the end

bitflipgremlin 的头像
bitflipgremlin1 个月前

and how would doing the RL in panama change any of this

Arip 的头像
Arip1 个月前

Electricity is cheap in Panama :)

DRAMYYDS (全网吹芯第一人) 的头像
DRAMYYDS (全网吹芯第一人)1 个月前

Ilya did not invent GPT-4 or GPT-3, it was work done by other people in OpenAI

Robin | Poker x AI 的头像
Robin | Poker x AI1 个月前

Replacing sum with max plus turns the objective into a clear best path EV, which explains the 16% lift you report

Coffee Cat 的头像
Coffee Cat1 个月前

Summing probability over successes means two half-right traces outweigh one full-right one. Max fits pass@1 but kills gradient on near-misses, which teach the most. Fix is a temperature'd softmax, not a hard max.

相关视频