Introducing FlexOlmo, a new paradigm for language model training that enables the co-development of AI through data collaboration. 🧵

FlexOlmo allows data owners to contribute to the development of language models without giving up control of their data. There’s no need to share raw data directly, and contributors can decide when their data is active in the model. 🔒

FlexOlmo employs a mixture-of-experts (MoE) architecture. Each expert is trained independently on local datasets and later integrated into an MoE. This allows data owners to contribute asynchronously without sharing their data while providing strong guarantees for data opt-out.

In our experiments, FlexOlmo often matches or exceeds the performance of specialized experts on their respective tasks. Notably, it even achieves performance very close to an upper-bound reference model trained on all combined public and closed datasets. 📈

Data owners who want to benefit from AI, but are hesitant to share their raw data or hand over control of it to a third party, can now participate without compromising the things they value.

✍️ Check out our blog: 📝 Read the paper: 💻 Visit the GitHub repo: ⬆️ See the model on Hugging Face:

A very interesting way to train and use a mixture of experts! Have created a nio for anyone who wants to quickly understand the technical paper:

Looks very interesting. A new paradigm

Data as "fuel" is so true! 🔥 How does FlexOlmo handle potential biases introduced by individual data contributors? Is there a mechanism for auditing data contributions for fairness? 🤔

Looks interesting!

Co-development through data collaboration... so does this mean the model learns to negotiate with datasets? 🤔 Intriguing approach, but what's the strategy for adversarial data contributors? 👀

@Eli1629761 @DefenderOfBasic - have either of you checked this out?

Sounds nice. Let's SEE

Woah super cool

Brilliant!