Our customers needed a better base model <10B parameters. We spent the last 5 months building one. I'm delighted to share a preview of our first Arcee Foundation Model: AFM-4.5B-Preview.

Existing models forced a choice: performance vs. compliance vs. future-proofing. None fully addressed the needs of modern orgs across regulated and resource-constrained environments.

AFM-4.5B is designed to: Run efficiently on modest hardware Meet Western regulatory standards Outperform stagnating offerings in the 3–10B space It’s not a patchwork: it’s a clean-slate model trained for today’s enterprise use cases.

We teamed up with @datologyai to build what we believe is the strongest pretraining corpus in the world—and I truly think we nailed it. Their team was absolutely key to the model’s success. We started with ~23T tokens of high-quality data and distilled it down to 6.58T through even more rigorous filtering.

Mid and post-training were key to performance: we used high-impact datasets, MergeKit for checkpoint merging, YaRN to extend context to 65,536 tokens, supervised fine-tuning for alignment, and RL + KTO for factual accuracy.

Our early results are extremely promising, and we wanted to get a preview out to collect feedback and inform the final release. AFM-4.5B and the base model will be released openly under cc-by-nc license in early July. Far more permissive models are in the works.

Since we're all-in on the Arcee family, we're open-sourcing our previously closed models: Virtuoso-Large (Qwen2.5-72B): Beats early GPT-4o checkpoints, released under Qwen's license SuperNova: Distilled from Llama-405B to Llama-3-70B, under the Llama-3 license Caller: Enterprise tool-use model (Qwen2.5-32B), Apache 2.0 license

Read our high level and technical blogs here:

Play with the model exclusively through @togethercompute here:

And find Virtuoso, SuperNova and Caller on our Huggingface page:

I could not be more proud of the team. In particular, @stochasticchasm @FernandoNetoAi @chargoddard. They put their heart and soul into this, and to reach the frontier of SLM models in our first run is a testament to their commitment, drive and passion.