Introducing Tahoe-x1 (Tx1) by Tahoe. A 3-billion-parameter, single-cell foundation model that learns unified representations of genes, cells, and drugs, achieving state-of-the-art performance across cancer-relevant cell biology benchmarks, open-sourced on Hugging Face. 🧵

2/ The past years have been defined by scaling laws: more compute & data, better models. This has held true for language and protein modeling. But does it also hold true for systems biology; for models that learn how cells & genes function across contexts and drug perturbations?

3/ Two barriers have stood in the way of answering that question: Lack of large, diverse single-cell data and Lack of compute-efficient models to enable search for better architectures and designs at billion-parameter scales.

4/ With Tahoe-100M, we created the largest perturbation dataset ever assembled to address the first barrier. And we are proud that it has been downloaded nearly 200K times in the last few months, a first for a biological dataset of this scale.

5/ Now we’re introducing Tahoe-x1 to tackle the second barrier. Tx1 is the first billion-parameter, compute-efficient foundation model trained on perturbation-rich single-cell data. And it is fully open-source with open weights.

6/ Tx1 is a Transformer (scGPT-inspired self-supervised objective) that stays practical to train at 3B parameter scale, enabling empirical search for optimal architectures and hyperparameters for modeling cells. It is 3-30x more compute efficient than other cell state models.

7/ To make it so, we borrowed the best tricks from LMs (FlashAttention v2, FSDP, Dataset streaming, Mixed-precision + multi-node scaling). But even cooler: We improved the attention operation at the heart of these models.

8/ In earlier Tx1 models we built  a Triton-optimized bias-matrix trick for masking, but the final design goes further: leveraging fully dense attention with FlashAttention v2 , simpler, faster, and still cutting GPU memory use by ~10×.

9/ We built Tx1 both not only as a starting point for exploration but as a benchmark for the field. We designed new benchmarks to evaluate its performance in cancer-relevant discovery and translational tasks.

10/ Tx1 achieves SOTA on key discovery tasks. It outperforms other models (and for the first time matches with linear baselines) in predicting gene essentiality as measured by the landmark DepMap dataset, a key piece of data in identifying subtype-specific targets.

11/ Similarly, it does better than other models in inferring hallmark oncogenic programs, another benchmark we designed leveraging MSigDB, to evaluate the model’s ability to identify core signatures of tumor progression.

12/ Tx1 also moves us closer to in silico prediction of cell response to drug & other perturbations, with the aim to ultimately generalize to in silico clinical trials. Combined with post-training frameworks, it predicts drug responses in unseen cell types and patient contexts.

13/ It further provides evidence that better embeddings generated from Tx1 compared to other models improve zero-shot performance to generalize in entirely new contexts and perturbations, confirming earlier results

14/ We want people to put Tx1 to test. We have built a tool to help you generate embeddings and visualize them for your own single-cell transcriptomic data.

15/ Progress will come not from one model, but from trying hundreds of designs and iterating on how best to represent the cell. We built Tx1 for that. That’s why we are open sourcing it - checkpoints, training code, evaluation workflows. Let's build! 🤗

Last/ Hats off to the superteam that made this happen, @shreshth_gandhi who led the work; Farnoosh Javadi, @vallens, and Umair Khan for their fantastic work, @MattG_Jones for his thoughtful contributions, @BoWang87 and @HAOTIANCUI1 for early conversations and @amerigo_81, @genophoria, @iamjohnnyyu for supervision!

@tahoe_ai @huggingface congratulations @nalidoust ! super exciting!

@tahoe_ai @huggingface Thank you sir 💙

@tahoe_ai @huggingface very very cool!

@tahoe_ai @huggingface Thanks @ClementDelangue 💙🤗

@tahoe_ai @huggingface Congratulations 👏👏 supercool project

@tahoe_ai @huggingface Congratulations @tahoe_ai team! Can't wait to take this for a spin.

@tahoe_ai @huggingface 🚀🚀

@tahoe_ai @huggingface Very impressive stuff!

@tahoe_ai @huggingface Thanks Carl!

@tahoe_ai @huggingface Very very cool. Keep going.

@tahoe_ai @huggingface Thank you Dan!

@tahoe_ai @huggingface Super cool!

@tahoe_ai @huggingface Thanks @jaanakprashar

@tahoe_ai @huggingface Congrats Nima + team!

@tahoe_ai @huggingface vvv interesting. addressing 'no internet for biology' problem

@tahoe_ai @huggingface Exciting breakthrough in cancer research! Amazing work.

@tahoe_ai @huggingface Open-sourcing the model on Hugging Face is a great move.

@tahoe_ai @huggingface Amazing work! Tabrik migam!

@tahoe_ai @huggingface 💙

@tahoe_ai @huggingface Congratulations. Great achievement

@tahoe_ai @huggingface noble work

@tahoe_ai @huggingface Impressive breakthrough! Excited to see its impact.

@tahoe_ai @huggingface super cool

@tahoe_ai @huggingface Thanks Zaais!

@tahoe_ai @huggingface @DeryaTR_ would love your take on this

@tahoe_ai @huggingface @DeryaTR_

@tahoe_ai @huggingface I love how Tx1’s open approach mirrors Dubai's biotech scene-collaboration like this speeds up real breakthroughs. 🚀

@tahoe_ai @huggingface Super excited to give this a go! Congrats on the launch

@tahoe_ai @huggingface 💙 tell us what you think!

@tahoe_ai @huggingface Understood (It is going to become very difficult to share appreciation and also caution up this growth curve) Can’t be everywhere all at once… but am going try regardless. Thank you Do great things for all Be careful please ❤️

@tahoe_ai @huggingface <pulls out ye ole Elon keyboard>Wow

@tahoe_ai @huggingface This is HUGE for bio-AI! The potential for Tx1 to accelerate drug discovery and cancer research is mind-blowing. Open-sourcing it on Hugging Face is the cherry on top.

@tahoe_ai @huggingface The real breakthrough isn’t performance on benchmarks it’s who controls the biological operating system

@tahoe_ai @huggingface Where I can use it online?

@tahoe_ai @huggingface Who designed your website Nima

@tahoe_ai @huggingface Damn

@tahoe_ai @huggingface Tahoe-x1’s unified modeling could be game-changing for single-cell analysis. Have you seen any early community applications or unexpected results since open-sourcing on HuggingFace?

@tahoe_ai @huggingface @ak_therich

@tahoe_ai @huggingface cool

@tahoe_ai @huggingface Congratulations ! If you want collab, go DM 😁

@tahoe_ai @huggingface lol "Ex-Rigetti"

@tahoe_ai @huggingface Congratulations 👏🏽🎉, I'm excited for the future of biology with Ai

@tahoe_ai @huggingface @SterlingCooley @BrianRoemmele