We previously shared our research on Layer Skip, an end-to-end solution for accelerating LLMs from researchers at Meta FAIR. It achieves this by executing a subset of an LLM’s layers and utilizing subsequent layers for verification and correction. We’re now releasing inference code and fine-tuned checkpoints for this work.... Model weights on Hugging Face ➡️ More details ➡️ We hope that releasing this work will open up new areas of experimentation and innovative new research in optimization and interpretability.show more

The paper explores ways to speed up the inference of large language models (LLMs) without significant loss in accuracy. LLMs are computationally expensive and have high financial and energy costs when deployed on GPU servers. The authors aim to address this challenge. The authors evaluate their approach on various tasks and model sizes. They show that their training recipe leads to higher accuracy in earlier layers compared to the baseline. They also demonstrate speedups of up to 2.16x on summarization, coding, and semantic parsing tasks using the self-speculative decoding approach. full paper:

can you guys please add `playsinline` to the video elements so it’s not a nightmare to scroll on mobile

Gonna be funky when an older model displaces a newer model on the leaderboards. “Grandpa Dense and Grandma MoE 8B appear to have displaced 405B” - LLM Commentators.

The release of Layer Skip is a significant step forward in optimizing LLM performance! By allowing for early exits and subsequent verification, it not only enhances efficiency but also opens the door for deeper explorations into model interpretability. Can't wait to see how the community leverages this technology to push boundaries further! ♥️

Great to see these optimizations for LLMs being shared! Tools like @kaibanjs make experimenting with these advancements more accessible for JavaScript developers 🚀

Poor man's MoE ?

@svpino - related to the dynamic models I mentioned

I had an error with the imapct grant application. I've been tring to get in touch. Submitable loaded slowly or crashed and I couldn't submit. Sent ticket to submittable. I first sent it on time by the deadline. I want to be evaluated please @Meta

Keep up the good work 💪

please add 3.1 70b 🥹