Video wird geladen...
Video konnte nicht geladen werden
fun weekend project: AutoJev. i was curious to see if i could train a competitive Jev-like model completely autonomously with a swarm of agents using our internal system. turns out you can get quite far! some details: - gave the swarm a devbox with an h200 gpu - the... show more
25,932 Aufrufe • vor 2 Tagen •via X (Twitter)
19 Kommentare

damn that compute budget, sparkling envy

super cool! how did the agents decide RL wasn't working? seemed like lots of the R&D swarms focused on curating the dataset likely mainly for SFT but i didn't notice the eval cluster growing much over time - wonder if RL got bottlenecked there or something else went wrong?

the part that surprised me is the agents figured out RL was not working and fell back to SFT on their own. that is the real demo here, not the benchmark numbers.

3k for a single project is so sick 🤯

jev-compatible api is what I'd test first. I ran kev-4b vs jev-1.13 on 59 versions of one privacy policy (236 labeled cells): at 0.85 both made 0 errors, kev just abstained a lot more. happy to run AutoJev on the same cells once inference is up. harness:

Giving the swarm a single devbox with one H200 is the interesting constraint here. Did the agents serialize on GPU time, or did you end up needing a queue/lease so two of them didn't stomp the same training run?

will you make the dataset public ?

The real result isn't the model, it's the method selection: the swarm noticed RL wasn't working and switched to SFT on its own. Research taste was supposed to be the hard part to automate.

training your own jev-like model, you might like this. been working the other side of it, small judge steering a bigger model's reasoning

The revealing result is that the swarm got farther with synthetic-data SFT than RL—did the agents spend more of their budget on data curation than on the training loop itself?

that’s super cool

a year ago this needed a research team. now it's one person, an agent swarm, and a weekend. the cost of replicating a frontier idea is collapsing to compute plus curiosity

That is a very interesting graphic. What SaaS do you use for visualization and dashboards?

Yeah. High quality synthetic data worked for us too. But in most cases we are seeing it regress back to mere chance

The fallback to SFT is the part I find most encouraging here. Switching to good synthetic data when RL wasn't working is closer to the research judgment I'd want from agents than just executing a fixed training recipe.

But what about the pink dude?

Free. Local. Private. Own your Ai brain. Laya > Jev by miles

ok.. it looks cool... but... is it correct?

"turns out you can get quite far" — one person, a devbox with an h200, and an agent swarm that ditched rl for sft on its own. if a weekend project can produce a competitive jev-like model, what's the point of the other four weekdays?
