Can LLMs simulate biology? We tested frontier models on real lab experiments: predict how cells respond to drugs, gene knockdowns, and immune signals. Even the best get only part of the response right. Lots of room to improve. Introducing BioSimBench, built on NVIDIA AI. 🧵

The task: given a cell and a drug, gene knockdown, or immune signal, predict which genes go up and down. Five tracks, one leaderboard.

Biological simulation can be learned. We trained a 27B model in BioSimGym, our RL environments for biology. It outscores the best frontier model on every BioSimBench task.

Built on NVIDIA. RAPIDS made the pipeline behind every task at least 4x faster than Scanpy, and up to 14.4x on single steps. Our models were trained on NVIDIA Blackwell GPUs. BioSimGym follows NeMo RL interfaces. @NVIDIAHealth @NVIDIAAI

Why simulate? Drugs, genes, and cell contexts create far more experiments than any lab can run. A co-scientist reasons about mechanisms. A simulator answers a different question: in this cell, after this intervention, what will the experiment measure?

Which model should we test next, and which experiment would you simulate? Submit a model or train in BioSimGym:

@NVIDIAAI Finally, a biology benchmark built on real lab data instead of textbook questions

@NVIDIAAI This is a great step for future AI research

CellType's 27B model, trained in BioSimGym, outperforms top frontier models by scoring higher on various tasks, thanks to efficient data processing with NVIDIA RAPIDS-singlecell and training with NVIDIA NeMo RL. Real experimental data is proving more effective than sheer scale! Great job!

@NVIDIAAI the interesting test is whether it predicts responses for drugs it never saw during training

@NVIDIAAI Which type of experiment was hardest for the models, drugs or knockdowns?

@NVIDIAAI Real progress. A 27B model trained in BioSimGym outperforming general frontier models in cellular response simulation is huge. Predicting cell reactions to drugs or signals before wet-lab tests will save massive time and trial-and-error. Excited to see more models tested on this.

@NVIDIAAI You just gave biology a gym membership 🤡

@NVIDIAAI Exciting to see how far AI can go in predicting real cellular behaviour. Lots of potential ahead!

@NVIDIAAI The next breakthrough in AI for science will come from better validation, not simply bigger models. Exciting work! 🔥

@NVIDIAAI built with nvidia too?? this is a serious launch

@NVIDIAAI finally a bio benchmark that measures something that matters in the lab

@NVIDIAAI This is where AI in biology starts getting really interesting. Less guessing, more testing.

@NVIDIAAI NVIDIA Blackwell working while everyone sleeps 😭 The night shift has a new favourite

@NVIDIAAI bookmarking this, the next year of AI for science is gonna be wild no cap

@NVIDIAAI This is the kind of benchmark AI actually needs. Less confidence, more scientific accuracy. Curious which models failed the hardest.

@NVIDIAAI How well do models generalize to unseen experiments?

@NVIDIAAI BioSimBench feels like a useful measuring stick for the next wave of biology-focused models.

@NVIDIAAI Exciting step toward understanding biology computationally. The gap between prediction and reality shows just how much potential there is for AI to advance scientific discovery.

@NVIDIAAI Exciting times for AI and life sciences!!

@NVIDIAAI Predicting which genes change after a drug treatment sounds like an incredibly valuable use of AI

@NVIDIAAI Real world testing like this is exactly what AI needs. Excited to see what researchers discover with this.

@NVIDIAAI Your leaderboard needs no explanation in a room full of biologists 👏

@NVIDIAAI Love this testing LLMs against real lab experiments is the benchmark biology actually needed!

@NVIDIAAI A fascinating benchmark that highlights both the potential and current limitations of AI in biological research.

@NVIDIAAI The gap between biological knowledge and biological prediction is fascinating. Would love to see which experimental conditions consistently break the strongest models.

@NVIDIAAI predicting cellular responses is a completely different beast than text or code, love seeing benchmarks tied directly to real lab data

@NVIDIAAI That’s impressive

@NVIDIAAI Real biology is the ultimate benchmark. This is exciting!

@NVIDIAAI Promising research with huge potential ahead

@NVIDIAAI BioSimBench is a clean reality check. Excited to see specialized models close the gap.

@NVIDIAAI love that they're honest about the results, "lots of room to improve" is refreshing fr