Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

We present Tropical Reinforcement Learning. Standard RL for LLMs cannot learn compositions effectively, for a fundamental reason: its algebra is not optimal. It sums the probabilities of every successful answer, measuring how often a model succeeds, not which solution succeeds.

43,127 Aufrufe • vor 1 Monat •via X (Twitter)

22 Kommentare

Profilbild von Arip
Aripvor 1 Monat

Our fix: change the algebra. Instead of summing over solutions, take the max → the tropical (max-plus) semiring. Each state's value is now the log-prob of its best verified solution, with the path itself. Good prefixes and suffixes from different rollouts can be joined.

Profilbild von Arip
Aripvor 1 Monat

Results: TROPIC beats the strongest on-policy baselines by up to 16 percentage points across four benchmarks. Big picture: changing the algebra of RL, not just its estimators, can make LLMs much better at compositional reasoning.

Profilbild von Arip
Aripvor 1 Monat

Paper: Code:

Profilbild von Swaroop Nath
Swaroop Nathvor 1 Monat

This is insane clickbait 🙂 I thought SSI released something

Profilbild von Arip
Aripvor 1 Monat

Sorry, I didn’t even think someone would see it that way.

Profilbild von Swaroop Nath
Swaroop Nathvor 1 Monat

It's no problem! Nice work, tho!

Profilbild von Joshua V. Dillon
Joshua V. Dillonvor 1 Monat

Very similar to our agentic discovery on sudoku/arc. Although our focus was max likelihood not RL, the idea is the same: backprop through the winning head but transfer state to losers too so rich don't get richer.

Profilbild von Ryan McCormick
Ryan McCormickvor 1 Monat

This is exactly right. Well trained means to represent the underlying algebra precisely.

Profilbild von Saksham Jain
Saksham Jainvor 1 Monat

if this holds, a lot of the pass@k shrinkage people pin on rl is really just the sum being the wrong thing to optimize

Profilbild von Arip
Aripvor 1 Monat

More results on pass@k soon.

Profilbild von Saksham Jain
Saksham Jainvor 1 Monat

looking forward to it. does the gap keep widening as k grows or does it level off past some point?

Profilbild von Catch Vartan
Catch Vartanvor 1 Monat

Fun fact: the tropical semiring was named such because it came from Brazil and the French mathematicians who saw just thought it was wild they were getting mathematics out of Brazil.

Profilbild von Arip
Aripvor 1 Monat

Yeah, max-plus is used more often, but I prefer tropical

Profilbild von Henrique Dalposso
Henrique Dalpossovor 1 Monat

It makes the LLM go bananas

Profilbild von Arip
Aripvor 1 Monat

😂😂

Profilbild von Greer
Greervor 1 Monat

Finally, congrats on the release! 🎉

Profilbild von Carter Leffen
Carter Leffenvor 1 Monat

@tairov you had me at Ilya and had me through the end

Profilbild von bitflipgremlin
bitflipgremlinvor 1 Monat

and how would doing the RL in panama change any of this

Profilbild von Arip
Aripvor 1 Monat

Electricity is cheap in Panama :)

Profilbild von DRAMYYDS (全网吹芯第一人)
DRAMYYDS (全网吹芯第一人)vor 1 Monat

Ilya did not invent GPT-4 or GPT-3, it was work done by other people in OpenAI

Profilbild von Robin | Poker x AI
Robin | Poker x AIvor 1 Monat

Replacing sum with max plus turns the objective into a clear best path EV, which explains the 16% lift you report

Profilbild von Coffee Cat
Coffee Catvor 1 Monat

Summing probability over successes means two half-right traces outweigh one full-right one. Max fits pass@1 but kills gradient on near-misses, which teach the most. Fix is a temperature'd softmax, not a hard max.

Ähnliche Videos