Jeff Dean's new company already has competition. Jeff Dean's Discovery Loop targets one key bottleneck with research today, i.e, research is human-intensive and runs one step at a time. His argument is that automating the full cycle raises both the count and the quality of experiments, starting with ML... research itself. Transformer Lab's Primus is built around a similar premise, with the research question as the only thing a human provides. It runs for roughly a day and returns a complete paper. The loop runs in six stages: - Literature review: reads and synthesizes existing work before designing anything - Experiment design: scopes the study, picks models, datasets, and metrics - Provisioning: brings up GPUs and schedules the jobs - Execution: runs training and evaluation, catches failures, and retries - Analysis: checks results against the design it committed to - Writeup: produces the paper with methodology, findings, and caveats Each stage on its own is usually a routine task, since agents already do all of them. But holding them together across a day of wall-clock time wasn't solved yet, since a crash after a few hours invalidates everything after it. That is a scheduling and state problem more than a model problem. I tested Primus on a quantization research question, asking it to compare INT8 and FP16 on a small open-source vision model and pick the better tradeoff for consumer GPU deployment. The video below shows the run and the paper it came back with. Try it yourself here: In my own run, as it worked, it caught two problems on its own. The image set it first downloaded was a compressed copy, and the compression was quietly inflating accuracy by about 2.9 points on all three models, so it went back and rebuilt every result on the original photos. It then noticed its own timing was unfair, since running the two precisions one after the other favoured INT8 at three of the four batch sizes, and switched to alternating them in short blocks. The result I would not have thought to look for is that INT8 costs ResNet-50 only 0.05 points of accuracy, which looks free, while quietly changing the answer on 4.3 percent of images. To know whether 4.3 percent was even a lot, it first built the same model twice in FP16 and measured how much two identical builds disagree with each other, which is 0.14 percent. Nobody has published that baseline, so there was never anything to compare against.show more

Akshay 🚀

284,616 subscribers

Teilen Herunterladen