Skip navigation
We're sharing a key step towards enabling Continual Learning at Applied Compute.
0:00 / 0:13

We're sharing a key step towards enabling Continual Learning at Applied Compute.

Applied Compute

@appliedcompute9.6K followers

Visit on X

39,498 views

We're sharing a key step towards enabling Continual Learning at Applied Compute. In On-Policy Self-Distillation, the teacher model supervises training with privileged information: a hint. The training signal depends heavily on hint quality, but manually refining hints across runs is impractical, and isolating their individual impact is difficult. We instead optimize hints automatically using efficient proxies for their eventual training value.

Comments

Comments from the original post on X (Twitter)

No comments yet