Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

1/n: 🔬We introduce NicheCompass (NC) a graph learning model to build spatial atlases across millions of cells, quantitatively characterize cell niches, infer cellular communication, and map new data into these atlases. NC can be used with various spatial technologies, ranging from cellular to subcellular resolution including even multimodal approaches.... show more

Mo Lotfollahi

12,999 subscribers

61,763 views • 2 years ago •via X (Twitter)

Science & Technology Education

Anya Rossi• Live Now

Private livecam show

10 Comments

Mo Lotfollahi2 years ago

2/n: The model draws inspiration from cellular signalling principles as an inductive bias. NC encodes cells (self) and their neighbourhoods (microenvironments) jointly, learning their relationships through the loss and architecture. It models cells as signal receivers from their microenvironment. NC learns interpretable cell representations, with each latent feature capturing the spatially localized activity of a gene program (GP). These GPs range from known ones encoding signalling to learning their targets and discovering new ones encoding spatial variability and potentially longer-range effects

Mo Lotfollahi2 years ago

3/n: Deciphering tissue architecture! Using NC, we can integrate distinct tissue samples, identify cell niches, and quantitatively characterize these based on learned GP activities. NC also enables the inference of cell-cell communication within and across niches on the GP level.

Mo Lotfollahi2 years ago

4/n: We systematically benchmarked NC across datasets, showing that the identified cell niches are anatomically meaningful and that the inferred representations and niches have high spatial and biological conservation, measured by a comprehensive suite of metrics.

Mo Lotfollahi2 years ago

5/n: To complement existing signalling knowledge, we design a mechanism to de novo learn gene programs (GPs). This enables effective use of data with limited or suboptimal gene panels, allowing us, for example, to identify spatially variable GPs in basal and luminal breast cancer niches.

Mo Lotfollahi2 years ago

6/n: We introduce spatial reference mapping using NC to efficiently contextualize a spatial query dataset within a reference atlas, enabling the seamless transfer of niche annotation (as opposed to cell type) and identification of novel niches present only in the query. We demonstrated this by mapping a new patient into a reference of non-small cell lung cancer patients, revealing a SPP1+ macrophage-dominated TME not present in other patients.

Mo Lotfollahi2 years ago

7/n: NC is extensible to spatial multi-omics data, including gene expression and chromatin accessibility. GPs then comprise genes and peaks based on chromosomal location or gene regulatory networks, enhancing characterization and unveiling niches previously undetectable.

Shila Ghazanfar2 years ago

Super exciting work!! Congratulations! 🎉🎉

Mo Lotfollahi2 years ago

Thanks Shila

Cecilia Lindgren2 years ago

Can’t wait to read this - how exciting ☺️⭐️

Mo Lotfollahi2 years ago

Thanks 🙏🏻

Related Videos

Excited to share our new work on building a multimodal atlas of human skin in health and inflammatory disease — a project I’m especially proud of, bringing together AI, high-throughput genomics, and clinical science to accelerate discovery. Over the past decade, single-cell genomics has transformed how we map cells in human tissues. But a major challenge remains: can we systematically decode how cells organize into functional niches in situ — including those invisible to standard histopathology? To address this, we integrated large-scale scRNA-seq, spatial transcriptomics, histopathology, and AI-driven modeling frameworks to build an in situ atlas of human skin across health and disease. Led by Lloyd Steele, an MD/PhD student working between Haniffa Lab and my lab at Wellcome Sanger Institute and Cambridge University . Another amazing collaboration with Muzz Haniffa, the mastermind behind the work as part of Human Cell Atlas. A key part of this study is that we didn’t build everything from scratch — we leveraged and combined AI methods that actually work! and showed how they can be used together to extract biological insight at scale. We used: • scArches to build and map into a reference scRNA-seq atlas of human skin: • NicheCompass to identify and characterize spatial niches: • MINT-Flow to extract microenvironment-induced cell states and gene programs: Together, these enabled an end-to-end workflow from atlas construction to spatial mapping, niche discovery, and cell state decoding. At scale, we integrated ~5 million cells and 100+ spatial sections, enabling a systematic view of tissue organization. Using this framework, we identified 26 niches in skin, including known histopathologic structures as well as hidden disease-associated niches not visible on H&E. Among the most striking findings were a resident memory T cell-rich sebaceous gland niche and a plasma cell-rich sweat gland niche, suggesting that appendageal structures act as active immunological microenvironments and may contribute to inflammatory memory and disease persistence. Importantly, this atlas is not just descriptive — it is usable. It can support mapping of new datasets, resolve finer cell types and niches, extract microenvironment-driven programs, and enable predictive analyses at scale. More broadly, this work shows what becomes possible when AI, spatial genomics, and atlas-scale data are integrated end-to-end: not just mapping tissues, but systematically decoding them. This was a massive collaboration, and I’m very grateful to the amazing scientists April Foster, Kenny Roberts, and Chloe Admane. Lloyd is an amazing scientist, and I’m especially excited for the community to see more of his work soon — stay tuned. The data and pre-trained models will be released soon. Preprint:

Excited to share our new work on building a multimodal atlas of human skin in health and inflammatory disease — a project I’m especially proud of, bringing together AI, high-throughput genomics, and clinical science to accelerate discovery. Over the past decade, single-cell genomics has transformed how we map cells in human tissues. But a major challenge remains: can we systematically decode how cells organize into functional niches in situ — including those invisible to standard histopathology? To address this, we integrated large-scale scRNA-seq, spatial transcriptomics, histopathology, and AI-driven modeling frameworks to build an in situ atlas of human skin across health and disease. Led by Lloyd Steele, an MD/PhD student working between Haniffa Lab and my lab at Wellcome Sanger Institute and Cambridge University . Another amazing collaboration with Muzz Haniffa, the mastermind behind the work as part of Human Cell Atlas. A key part of this study is that we didn’t build everything from scratch — we leveraged and combined AI methods that actually work! and showed how they can be used together to extract biological insight at scale. We used: • scArches to build and map into a reference scRNA-seq atlas of human skin: • NicheCompass to identify and characterize spatial niches: • MINT-Flow to extract microenvironment-induced cell states and gene programs: Together, these enabled an end-to-end workflow from atlas construction to spatial mapping, niche discovery, and cell state decoding. At scale, we integrated ~5 million cells and 100+ spatial sections, enabling a systematic view of tissue organization. Using this framework, we identified 26 niches in skin, including known histopathologic structures as well as hidden disease-associated niches not visible on H&E. Among the most striking findings were a resident memory T cell-rich sebaceous gland niche and a plasma cell-rich sweat gland niche, suggesting that appendageal structures act as active immunological microenvironments and may contribute to inflammatory memory and disease persistence. Importantly, this atlas is not just descriptive — it is usable. It can support mapping of new datasets, resolve finer cell types and niches, extract microenvironment-driven programs, and enable predictive analyses at scale. More broadly, this work shows what becomes possible when AI, spatial genomics, and atlas-scale data are integrated end-to-end: not just mapping tissues, but systematically decoding them. This was a massive collaboration, and I’m very grateful to the amazing scientists April Foster, Kenny Roberts, and Chloe Admane. Lloyd is an amazing scientist, and I’m especially excited for the community to see more of his work soon — stay tuned. The data and pre-trained models will be released soon. Preprint:

Mo Lotfollahi

11,759 views • 3 months ago

How does an embryo reliably "compute" its form - "cell by cell" - using only local interactions and mechanics, yet produce a precise global body plan? I’m excited to share our Nature Methods paper "MultiCell: geometric learning in multicellular development", presenting #AIxBiology research led by Haiqian Yang and the result of a great collaboration with Ming Guo, George Roy, Tomer Stern, Anh Nguyen and Dapeng Bi. A long-standing challenge in developmental biology is to predict how thousands of cells collectively self-organize as tissues fold, divide, and rearrange. In MultiCell, we represent a developing embryo as a dual graph that unifies two complementary views of tissue mechanics with single-cell resolution: cells as moving points (granular) and cells as a connected foam (junction network). This lets the model learn dynamics from both geometry and cell–cell connectivity. On whole-embryo 4D light-sheet movies of Drosophila gastrulation (~5,000 cells), our model predicts key cell behaviors and the timing of events, including junction loss, rearrangements, and divisions with high accuracy, at single-cell resolution. Beyond prediction, the same representation supports robust time alignment across embryos and offers interpretable activation maps that highlight the morphogenetic "drivers" of development. The broader goal is a foundation for cell-by-cell forecasting in more complex tissues, and eventually for detecting subtle dynamical signatures of disease. Kudos to the team for this inspiring collaboration with brilliant researchers to push the boundary of AI for biology! Citation: Yang, H., Roy, G., Nguyen, A.Q., Buehler, M.J., et al. MultiCell: geometric learning in multicellular development. Nature Methods (2025), DOI: 10.1038/s41592-025-02983-x Code/data links are in the manuscript.

How does an embryo reliably "compute" its form - "cell by cell" - using only local interactions and mechanics, yet produce a precise global body plan? I’m excited to share our Nature Methods paper "MultiCell: geometric learning in multicellular development", presenting #AIxBiology research led by Haiqian Yang and the result of a great collaboration with Ming Guo, George Roy, Tomer Stern, Anh Nguyen and Dapeng Bi. A long-standing challenge in developmental biology is to predict how thousands of cells collectively self-organize as tissues fold, divide, and rearrange. In MultiCell, we represent a developing embryo as a dual graph that unifies two complementary views of tissue mechanics with single-cell resolution: cells as moving points (granular) and cells as a connected foam (junction network). This lets the model learn dynamics from both geometry and cell–cell connectivity. On whole-embryo 4D light-sheet movies of Drosophila gastrulation (~5,000 cells), our model predicts key cell behaviors and the timing of events, including junction loss, rearrangements, and divisions with high accuracy, at single-cell resolution. Beyond prediction, the same representation supports robust time alignment across embryos and offers interpretable activation maps that highlight the morphogenetic "drivers" of development. The broader goal is a foundation for cell-by-cell forecasting in more complex tissues, and eventually for detecting subtle dynamical signatures of disease. Kudos to the team for this inspiring collaboration with brilliant researchers to push the boundary of AI for biology! Citation: Yang, H., Roy, G., Nguyen, A.Q., Buehler, M.J., et al. MultiCell: geometric learning in multicellular development. Nature Methods (2025), DOI: 10.1038/s41592-025-02983-x Code/data links are in the manuscript.

Markus J. Buehler

387,913 views • 6 months ago

Single-cell technologies now let us profile entire transcriptomes in individual cells. But how do we make sense of this complexity in a biologically meaningful way? Many methods summarise cells into a single embedding, but this often comes at the cost of interpretability, especially when multiple gene programs are active at once. We developed Tripso, a self-supervised transformer model that represents cells through multiple gene program-specific embeddings, while also uncovering new programs directly from the data. Instead of collapsing biology into a single vector, Tripso decomposes cell state into multiple representations, each reflecting a different gene program. We explored this across multiple systems. In human hematopoiesis, spanning development to aging, Tripso identified distinct age-associated program activity, including stronger JAK-STAT signalling in early life and dynamic IKZF1-related changes during B cell maturation. By comparing in vitro culture conditions with in vivo hematopoietic stem cell states, Tripso suggested that targeting the SEC61 translocon could enhance stem cell maintenance ex vivo, a prediction that we subsequently validated experimentally. In parallel, we identified a previously uncharacterised tissue-resident memory T-cell program associated with atopic dermatitis and mapped it to distinct spatial immune niches Together, these results show how modelling cells through gene programs can lead to interpretable and experimentally testable insights. More broadly, this work points toward a more interpretable and biologically grounded models of cell state. As single-cell datasets continue to grow, we hope approaches like Tripso will help bridge the gap between data-driven representations and biological insight. This work wouldn’t have been possible without the contributions of an amazing team. Thank you to co-first authors Marie, Tomoya Isobe, Amirhosein Vahidi, Carlo Leonardi, and everyone from roser's Lab, Haniffa Lab, Nicola Wilson and Bertie Gottgens's Lab, bringing together expertise across Cambridge Stem Cell Institute, Open Targets, Wellcome Sanger Institute and Cambridge University. Marie is one of the very best PhD students I have ever supervised. She is truly a force of nature, exceptionally resourceful, deeply innovative, and one of the most impressive scientists I have worked with. I am immensely proud of her and all that she has accomplished. As she begins her internship at Genentech , I have no doubt she will do amazing work there and continue to make her mark. paper: code:

Mo Lotfollahi

22,402 views • 3 months ago

We are thrilled to share that our first paper from my new lab, Spateo ( for spatiotemporal modeling of molecular holograms, is now online in Cell: Spateo is a comprehensive analytical framework for 3D whole-embryo spatiotemporal modeling. Its advanced features include: • 3D alignment and reconstruction at the whole-mouse-embryo scale (see the animation). • 3D spatial domain digitization and cell-cell communication analysis to understand spatial gene expression gradients and both inter- and intracellular communication. • 3D morphometric and volumetric analyses along with 3D morphogenesis vector field modeling to quantify dynamics such as surface area, volume, and cell density across organs, and to dissect the interplay between morphogenesis factors and cell migration. • A “Google Earth”-like browser, Spateo-viewer ( and for interactive and intuitive exploration of 3D spatial data. • Additional features, such as RNA signal-based single-cell segmentation. We are also honored that Nature “News and Views” has highlighted this work as well: This is really an amazing outcome after two years' heroic revision process that rewrite the entire paper using a new data ( for whole mouse embryos.

We are thrilled to share that our first paper from my new lab, Spateo ( for spatiotemporal modeling of molecular holograms, is now online in Cell: Spateo is a comprehensive analytical framework for 3D whole-embryo spatiotemporal modeling. Its advanced features include: • 3D alignment and reconstruction at the whole-mouse-embryo scale (see the animation). • 3D spatial domain digitization and cell-cell communication analysis to understand spatial gene expression gradients and both inter- and intracellular communication. • 3D morphometric and volumetric analyses along with 3D morphogenesis vector field modeling to quantify dynamics such as surface area, volume, and cell density across organs, and to dissect the interplay between morphogenesis factors and cell migration. • A “Google Earth”-like browser, Spateo-viewer ( and for interactive and intuitive exploration of 3D spatial data. • Additional features, such as RNA signal-based single-cell segmentation. We are also honored that Nature “News and Views” has highlighted this work as well: This is really an amazing outcome after two years' heroic revision process that rewrite the entire paper using a new data ( for whole mouse embryos.

evo-devo

58,343 views • 1 year ago

🚀 We are hiring! 𝗣𝗵𝗗 𝗦𝘁𝘂𝗱𝗲𝗻𝘁 𝗶𝗻 𝗖𝗔𝗥-𝗧 𝗖𝗲𝗹𝗹𝘂𝗹𝗮𝗿 𝗧𝗵𝗲𝗿𝗮𝗽𝘆 𝗮𝗻𝗱 𝗖𝗮𝗻𝗰𝗲𝗿 𝗡𝗲𝘂𝗿𝗼𝗶𝗺𝗺𝘂𝗻𝗼𝗹𝗼𝗴𝘆 We are looking for a passionate and motivated PhD student to join an exciting project focused on how nerve, immune and cancer cell interactions shape metastatic tumor behavior and resistance to cellular therapies, especially CAR-T cells. The project combines: • Whole-body tissue clearing • Light-sheet microscopy • wildDISCO-based immunolabeling • CAR-T and immune cell tracking in metastatic cancer • Spatial proteomics and 3D spatial biology • AI-based analysis of large-scale imaging datasets This is a great opportunity for someone with a background in cancer biology, immunology, neuroscience, biomedical sciences, biomedical engineering or a related field who wants to work at the interface of experimental biology, advanced imaging and AI. Apply here: Please share with excellent candidates who may be interested. #Hiring #PhDPosition #CARTcells #CellularTherapy #CancerResearch #Neuroimmunology #Immunotherapy #Metastasis #SpatialBiology #AIinBiology #Microscopy #LMUMunich #HelmholtzMunich #ErturkLab

🚀 We are hiring! 𝗣𝗵𝗗 𝗦𝘁𝘂𝗱𝗲𝗻𝘁 𝗶𝗻 𝗖𝗔𝗥-𝗧 𝗖𝗲𝗹𝗹𝘂𝗹𝗮𝗿 𝗧𝗵𝗲𝗿𝗮𝗽𝘆 𝗮𝗻𝗱 𝗖𝗮𝗻𝗰𝗲𝗿 𝗡𝗲𝘂𝗿𝗼𝗶𝗺𝗺𝘂𝗻𝗼𝗹𝗼𝗴𝘆 We are looking for a passionate and motivated PhD student to join an exciting project focused on how nerve, immune and cancer cell interactions shape metastatic tumor behavior and resistance to cellular therapies, especially CAR-T cells. The project combines: • Whole-body tissue clearing • Light-sheet microscopy • wildDISCO-based immunolabeling • CAR-T and immune cell tracking in metastatic cancer • Spatial proteomics and 3D spatial biology • AI-based analysis of large-scale imaging datasets This is a great opportunity for someone with a background in cancer biology, immunology, neuroscience, biomedical sciences, biomedical engineering or a related field who wants to work at the interface of experimental biology, advanced imaging and AI. Apply here: Please share with excellent candidates who may be interested. #Hiring #PhDPosition #CARTcells #CellularTherapy #CancerResearch #Neuroimmunology #Immunotherapy #Metastasis #SpatialBiology #AIinBiology #Microscopy #LMUMunich #HelmholtzMunich #ErturkLab

Ali Max Erturk

14,750 views • 2 months ago

Excited to share our new work. Over the past decade, single-cell genomics has transformed our ability to map cellular systems. But a major question remains: Can we predict how perturbations reshape cellular trajectories over time? In 2018, we first showed that it is possible to predict cellular responses to perturbations — ranging from disease signals to chemical treatments — even in unseen contexts. In 2022, we introduced CPA (MSB 2022; NeurIPS 2022), extending this idea to predict responses to unseen chemical and genetic perturbations, including their combinations. Since then, the field of perturbation modeling has grown enormously. The community has pushed the space forward with many creative ideas and powerful models. It’s exciting to see how fast things are moving — even though many fundamental challenges remain. One of the biggest is that cells are not static. They move through trajectories during development, immune responses, and disease. Yet most current models still predict perturbation effects within a single state, rather than how early perturbations propagate across future states and reshape downstream outcomes. To address this, we developed PerturbGen, a trajectory-aware generative AI model that predicts how genetic perturbations reshape downstream cellular states. Huge credit to the people who made this work possible. Thanks to co-first authors Kevin Ly, Adib Miraki, Tomoya Isobe, AmirHoss3in Vahidi, Delshad Vaghari & Anthony Rostron. Special recognition to Kevin Ly and Adib Miraki for driving this work over the finish line. Grateful for our outstanding collaborators from Haniffa Lab, Bertie Gottgens lab Gosia Trynka and many others — a true cross-institute effort across Cambridge Stem Cell Institute, Open Targets ,Wellcome Sanger Institute and Cambridge University.🎉 PerturbGen learns transcriptional dynamics across cellular trajectories. By introducing perturbations at an early source state, it can simulate how these effects propagate into future states along differentiation trajectories. Scaling this across genes enables the creation of dynamic in silico perturbation atlases — maps of how perturbations reshape biological trajectories over time. We explored this idea across three biological questions. First, in a human in vivo LPS immune challenge, PerturbGen predicted that perturbing a transient IL1B signal dampens downstream inflammatory programs in myeloid cells, with pathway changes reversing signatures observed in an independent IL-1β stimulation experiment. Second, in human hematopoiesis, PerturbGen predicted transcriptional responses to CRISPR transcription factor knockouts and enabled construction of perturbation atlases revealing lineage- and age-specific regulatory programs. These programs could also be linked to human genetics and blood diseases, including recapitulation of signatures associated with ETV6-related thrombocytopenia. Finally, we asked whether perturbation modeling could help improve complex tissue models. We built a dynamic perturbation atlas of human skin organoids to identify perturbations that could guideorganoid cells towardhuman fetal skin states. PerturbGen prioritized activation of Wnt signaling via GSK3β inhibition. Experimental validation confirmed the prediction: treatment with CHIR99021 induced stromal gene programs and shifted organoid fibroblasts toward transcriptional states observed in fetal skin stroma. Together, these results show how trajectory-aware perturbation modeling can connect gene perturbations to developmental programs, human genetics, disease mechanisms, and experimental interventions. More broadly, we think these point toward a future where single-cell atlases become predictive systems. As atlases expand across tissues, developmental windows, and modalities, models like PerturbGen could enable dynamic, virtual perturbation atlases— allowing us to simulate interventions, generate hypotheses, and design experiments before stepping into the lab. Preprint Code Excited to see how the community builds on this work.

Excited to share our new work. Over the past decade, single-cell genomics has transformed our ability to map cellular systems. But a major question remains: Can we predict how perturbations reshape cellular trajectories over time? In 2018, we first showed that it is possible to predict cellular responses to perturbations — ranging from disease signals to chemical treatments — even in unseen contexts. In 2022, we introduced CPA (MSB 2022; NeurIPS 2022), extending this idea to predict responses to unseen chemical and genetic perturbations, including their combinations. Since then, the field of perturbation modeling has grown enormously. The community has pushed the space forward with many creative ideas and powerful models. It’s exciting to see how fast things are moving — even though many fundamental challenges remain. One of the biggest is that cells are not static. They move through trajectories during development, immune responses, and disease. Yet most current models still predict perturbation effects within a single state, rather than how early perturbations propagate across future states and reshape downstream outcomes. To address this, we developed PerturbGen, a trajectory-aware generative AI model that predicts how genetic perturbations reshape downstream cellular states. Huge credit to the people who made this work possible. Thanks to co-first authors Kevin Ly, Adib Miraki, Tomoya Isobe, AmirHoss3in Vahidi, Delshad Vaghari & Anthony Rostron. Special recognition to Kevin Ly and Adib Miraki for driving this work over the finish line. Grateful for our outstanding collaborators from Haniffa Lab, Bertie Gottgens lab Gosia Trynka and many others — a true cross-institute effort across Cambridge Stem Cell Institute, Open Targets ,Wellcome Sanger Institute and Cambridge University.🎉 PerturbGen learns transcriptional dynamics across cellular trajectories. By introducing perturbations at an early source state, it can simulate how these effects propagate into future states along differentiation trajectories. Scaling this across genes enables the creation of dynamic in silico perturbation atlases — maps of how perturbations reshape biological trajectories over time. We explored this idea across three biological questions. First, in a human in vivo LPS immune challenge, PerturbGen predicted that perturbing a transient IL1B signal dampens downstream inflammatory programs in myeloid cells, with pathway changes reversing signatures observed in an independent IL-1β stimulation experiment. Second, in human hematopoiesis, PerturbGen predicted transcriptional responses to CRISPR transcription factor knockouts and enabled construction of perturbation atlases revealing lineage- and age-specific regulatory programs. These programs could also be linked to human genetics and blood diseases, including recapitulation of signatures associated with ETV6-related thrombocytopenia. Finally, we asked whether perturbation modeling could help improve complex tissue models. We built a dynamic perturbation atlas of human skin organoids to identify perturbations that could guideorganoid cells towardhuman fetal skin states. PerturbGen prioritized activation of Wnt signaling via GSK3β inhibition. Experimental validation confirmed the prediction: treatment with CHIR99021 induced stromal gene programs and shifted organoid fibroblasts toward transcriptional states observed in fetal skin stroma. Together, these results show how trajectory-aware perturbation modeling can connect gene perturbations to developmental programs, human genetics, disease mechanisms, and experimental interventions. More broadly, we think these point toward a future where single-cell atlases become predictive systems. As atlases expand across tissues, developmental windows, and modalities, models like PerturbGen could enable dynamic, virtual perturbation atlases— allowing us to simulate interventions, generate hypotheses, and design experiments before stepping into the lab. Preprint Code Excited to see how the community builds on this work.

Mo Lotfollahi

17,035 views • 4 months ago

🚀 Thrilled to announce #inTRACKtive: a web-based tool for exploring massive cell-tracking datasets, no software installation required! Just open your browser and dive into terabytes of developmental biology data. We used it to build the virtual of tracked embryonic development datasets 🐭🪰🪱🪲🐠, but you can also use it for your own data! 🔬 Preprint: Repository: This project was spear-headed by Teun Huijben, together with engineers Andrew Sweet and @aganders3 from Biohub @czbbiohub #CZBiohubSF #devbio #tracking #web #visualization 🧵1/n

🚀 Thrilled to announce #inTRACKtive: a web-based tool for exploring massive cell-tracking datasets, no software installation required! Just open your browser and dive into terabytes of developmental biology data. We used it to build the virtual of tracked embryonic development datasets 🐭🪰🪱🪲🐠, but you can also use it for your own data! 🔬 Preprint: Repository: This project was spear-headed by Teun Huijben, together with engineers Andrew Sweet and @aganders3 from Biohub @czbbiohub #CZBiohubSF #devbio #tracking #web #visualization 🧵1/n

Loïc A. Royer 💻🔬⚗️

17,680 views • 1 year ago

PantheonOS allows any biologist to perform complex data analyses of emerging single cell, multi-omics and spatial transcriptomics datasets end to end through AI agent and human collaboration. We are releasing six replays of use cases trajectories. Each "trajectory" is a complete end-to-end run, from prompt to analysis, to figures, and finally to report, and in fully inspectable and reproducible manner. See the six user cases in our PantheonOS Gallery: 1. 3D mouse embryo analysis: Tangram deconvolution and PyVista-based 3D visualization of E6 mouse embryo spatial transcriptomics data 2. 3D human fetal heart analysis: Spatial mapping of heart disease gene patterns using MERFISH data 3. Multi-omics spatial integration: Single cell multi-omics-to-spatial mapping with MOSCOT optimal transport 4. Gene panel design: 1000-plex immune-oncology MERFISH gene panel optimization 5. Cell segmentation benchmarking: Comparative evaluation of Cellpose-SAM, InstanSeg, StarDist, and other tools 6. Spatial disease biology: Ligand–receptor analysis of disease-associated tissue microenvironments We would love to hear how you can use PantheonOS for your research! Got an interesting agent run of your own and want to share? In the Pantheon UI, click Export Bundle (top-right of any chat) to package the full trajectory — chat history, code, figures, report — then submit it here:

PantheonOS allows any biologist to perform complex data analyses of emerging single cell, multi-omics and spatial transcriptomics datasets end to end through AI agent and human collaboration. We are releasing six replays of use cases trajectories. Each "trajectory" is a complete end-to-end run, from prompt to analysis, to figures, and finally to report, and in fully inspectable and reproducible manner. See the six user cases in our PantheonOS Gallery: 1. 3D mouse embryo analysis: Tangram deconvolution and PyVista-based 3D visualization of E6 mouse embryo spatial transcriptomics data 2. 3D human fetal heart analysis: Spatial mapping of heart disease gene patterns using MERFISH data 3. Multi-omics spatial integration: Single cell multi-omics-to-spatial mapping with MOSCOT optimal transport 4. Gene panel design: 1000-plex immune-oncology MERFISH gene panel optimization 5. Cell segmentation benchmarking: Comparative evaluation of Cellpose-SAM, InstanSeg, StarDist, and other tools 6. Spatial disease biology: Ligand–receptor analysis of disease-associated tissue microenvironments We would love to hear how you can use PantheonOS for your research! Got an interesting agent run of your own and want to share? In the Pantheon UI, click Export Bundle (top-right of any chat) to package the full trajectory — chat history, code, figures, report — then submit it here:

evo-devo

13,151 views • 2 months ago

🚀 Introducing scGPT-spatial! 🧬🌍 A game-changing spatial-omic foundation model, built on the powerful scGPT framework with MoE (mixture of experts) and continually pretrained on a massive 30 million spatial single-cell profiles! 🧠 What’s the challenge? Spatial transcriptomics is next-level complex—not only must we model single-cell/spot profiles, but we also need to capture intricate spatial relationships while handling diverse sequencing protocols (imaging-based vs. sequencing-based). 🔥 Why scGPT-spatial? ✨ A Spatial-omic Foundation Model with Continual Pretraining – Built on scGPT’s robust initialization, it unlocks spatial context in tissues. ✨ SpatialHuman30M Dataset – The largest curated dataset: 30M profiles from Visium, Visium HD, Xenium, and MERFISH across 821 slides. ✨ Revolutionary MoE Decoders – A cutting-edge Mixture of Experts (MoE) architecture for protocol-aware gene expression decoding. ✨ Spatially-Aware Training Strategy – A neighborhood-based masked reconstruction approach to capture complex cell-type colocalization. ✨ Multi-Modal & Multi-Slide Integration – Seamless clustering & spatial domain identification across slides and modalities. ✨ Cell-Type Deconvolution & Gene Imputation – Unlocks cross-resolution & cross-modality harmonization with fine-tuned embeddings. 📄 Read the preprint: 💻 Explore the code/weights: #SpatialTranscriptomics #SingleCell #AIResearch #MachineLearning #SpatialData Huge shoutout to the incredible PHD students Chloe (ChloeXWang) and Haotian (Haotian Cui) for leading this groundbreaking project! 🎉 Massive thanks to our amazing co-authors Andrew, Ronald, and Hani (Hani Goodarzi) from Arc Institute—this work wouldn't have been possible without you! 👏

🚀 Introducing scGPT-spatial! 🧬🌍 A game-changing spatial-omic foundation model, built on the powerful scGPT framework with MoE (mixture of experts) and continually pretrained on a massive 30 million spatial single-cell profiles! 🧠 What’s the challenge? Spatial transcriptomics is next-level complex—not only must we model single-cell/spot profiles, but we also need to capture intricate spatial relationships while handling diverse sequencing protocols (imaging-based vs. sequencing-based). 🔥 Why scGPT-spatial? ✨ A Spatial-omic Foundation Model with Continual Pretraining – Built on scGPT’s robust initialization, it unlocks spatial context in tissues. ✨ SpatialHuman30M Dataset – The largest curated dataset: 30M profiles from Visium, Visium HD, Xenium, and MERFISH across 821 slides. ✨ Revolutionary MoE Decoders – A cutting-edge Mixture of Experts (MoE) architecture for protocol-aware gene expression decoding. ✨ Spatially-Aware Training Strategy – A neighborhood-based masked reconstruction approach to capture complex cell-type colocalization. ✨ Multi-Modal & Multi-Slide Integration – Seamless clustering & spatial domain identification across slides and modalities. ✨ Cell-Type Deconvolution & Gene Imputation – Unlocks cross-resolution & cross-modality harmonization with fine-tuned embeddings. 📄 Read the preprint: 💻 Explore the code/weights: #SpatialTranscriptomics #SingleCell #AIResearch #MachineLearning #SpatialData Huge shoutout to the incredible PHD students Chloe (ChloeXWang) and Haotian (Haotian Cui) for leading this groundbreaking project! 🎉 Massive thanks to our amazing co-authors Andrew, Ronald, and Hani (Hani Goodarzi) from Arc Institute—this work wouldn't have been possible without you! 👏

Bo Wang

59,008 views • 1 year ago

📡🧵 Excited to release two #bioRxiv preprints! The first preprint is: "Zebrahub-Multiome: Uncovering Gene Regulatory Network Dynamics During Zebrafish Embryogenesis" We present a comprehensive time-resolved single-cell multiomic atlas of zebrafish development. Integrating scATAC-seq and scRNA-seq from 94,000+ cells, we map gene regulatory networks across six key stages of embryogenesis (10 to 24 hpf). A major accomplishment by YangJoon Kim co-led by Merlin Lange with experiments by Shruthi VijayKumar #Zebrahub-Multiome is a sequel to our transcriptomic and imaging atlas of zebrafish development. #zebrafish #development #ATACseq #cell #biology #sequencing Chan Zuckerberg Biohub Network #CZBiohubSF Biohub Note: We sent two preprints simultaneously yesterday, and this is the first one to come out... I will append to this thread when the second one is out...

📡🧵 Excited to release two #bioRxiv preprints! The first preprint is: "Zebrahub-Multiome: Uncovering Gene Regulatory Network Dynamics During Zebrafish Embryogenesis" We present a comprehensive time-resolved single-cell multiomic atlas of zebrafish development. Integrating scATAC-seq and scRNA-seq from 94,000+ cells, we map gene regulatory networks across six key stages of embryogenesis (10 to 24 hpf). A major accomplishment by YangJoon Kim co-led by Merlin Lange with experiments by Shruthi VijayKumar #Zebrahub-Multiome is a sequel to our transcriptomic and imaging atlas of zebrafish development. #zebrafish #development #ATACseq #cell #biology #sequencing Chan Zuckerberg Biohub Network #CZBiohubSF Biohub Note: We sent two preprints simultaneously yesterday, and this is the first one to come out... I will append to this thread when the second one is out...

Loïc A. Royer 💻🔬⚗️

39,773 views • 1 year ago

A new Nature paper from Johns Hopkins (by Prof. Lin Dingchang Lin ) just solved one of the hardest problems in biology: how do you record what every cell in a tissue experienced over time, not just what it looks like right now? The answer: GEMINI — Granularly Expanding Memory for Intracellular Narrative Integration. It works exactly like tree rings. Cells are genetically engineered to express a computationally designed protein assembly. As the assembly grows inside the cell, it captures cellular activity as fluorescent ring patterns — each ring a timestamp, each ring's properties encoding signal intensity. Look at a cross-section under a microscope and you can read the cell's history backward, with ~15-minute resolution. The key: cells build the recorder themselves. GEMINI doesn't interfere with normal function — it just quietly writes. What they demonstrated: In a full tumor xenograft, GEMINI captured every cancer cell's activity history across the entire tumor while it continued to grow normally. For the first time, researchers can look back and see how different regions of the same tumor responded differently to therapy over time — not snapshots, but film. In a mouse brain, GEMINI recorded neural activity dynamics without disrupting behavior, coordination, or memory. It could temporally resolve the history of a brain seizure. Why this matters: Every tool we have in biology gives you state — what the cell looks like now. Sequencing, imaging, proteomics — all snapshots. GEMINI gives you trajectory. It's the difference between a photograph and a video, applied to every cell in an organ simultaneously. The team is explicit that AI-based decoding tools will be central to reading GEMINI's output at whole-brain scale. This is the data layer that makes temporal single-cell atlases possible. Paper: Congratulations Dingchang Lin

A new Nature paper from Johns Hopkins (by Prof. Lin Dingchang Lin ) just solved one of the hardest problems in biology: how do you record what every cell in a tissue experienced over time, not just what it looks like right now? The answer: GEMINI — Granularly Expanding Memory for Intracellular Narrative Integration. It works exactly like tree rings. Cells are genetically engineered to express a computationally designed protein assembly. As the assembly grows inside the cell, it captures cellular activity as fluorescent ring patterns — each ring a timestamp, each ring's properties encoding signal intensity. Look at a cross-section under a microscope and you can read the cell's history backward, with ~15-minute resolution. The key: cells build the recorder themselves. GEMINI doesn't interfere with normal function — it just quietly writes. What they demonstrated: In a full tumor xenograft, GEMINI captured every cancer cell's activity history across the entire tumor while it continued to grow normally. For the first time, researchers can look back and see how different regions of the same tumor responded differently to therapy over time — not snapshots, but film. In a mouse brain, GEMINI recorded neural activity dynamics without disrupting behavior, coordination, or memory. It could temporally resolve the history of a brain seizure. Why this matters: Every tool we have in biology gives you state — what the cell looks like now. Sequencing, imaging, proteomics — all snapshots. GEMINI gives you trajectory. It's the difference between a photograph and a video, applied to every cell in an organ simultaneously. The team is explicit that AI-based decoding tools will be central to reading GEMINI's output at whole-brain scale. This is the data layer that makes temporal single-cell atlases possible. Paper: Congratulations Dingchang Lin

Bo Wang

85,095 views • 4 months ago

Dr. Patrick Soon-Shiong on the Cancer-Fighting Immune Breakthrough: Awakening the 450 Million-Year-Old Natural Killer Cell After decades of pioneering work since discovering the natural killer cell in the 1990s—a remarkable immune cell created by God and preserved through 450 million years—Dr. Patrick Soon-Shiong reveals an extraordinary breakthrough. This cell, unknown to many until the 1970s, serves as the body's first line of defense, designed to destroy cancer cells, infections like COVID, sepsis, bacteria, and fungus. Dr. Soon-Shiong emphasizes that this is not an invention, but an enlightenment: activating the body’s own natural killer cells to combat disease naturally, shifting away from toxic chemotherapy and radiation. With a single jab, they have unlocked the secret to empowering this primal cellular protector. This is evolution’s own weapon against illness, a game-changer that taps into ancient biology to revolutionize modern medicine. This insight reframes how we think about health and cancer treatment by harnessing the immune system's innate power, moving beyond harsh treatments, and embracing nature’s original disease-fighter from within. The discovery is a call to awaken the body’s own god-given defense system for a future of empowered healing.

Dr. Patrick Soon-Shiong on the Cancer-Fighting Immune Breakthrough: Awakening the 450 Million-Year-Old Natural Killer Cell After decades of pioneering work since discovering the natural killer cell in the 1990s—a remarkable immune cell created by God and preserved through 450 million years—Dr. Patrick Soon-Shiong reveals an extraordinary breakthrough. This cell, unknown to many until the 1970s, serves as the body's first line of defense, designed to destroy cancer cells, infections like COVID, sepsis, bacteria, and fungus. Dr. Soon-Shiong emphasizes that this is not an invention, but an enlightenment: activating the body’s own natural killer cells to combat disease naturally, shifting away from toxic chemotherapy and radiation. With a single jab, they have unlocked the secret to empowering this primal cellular protector. This is evolution’s own weapon against illness, a game-changer that taps into ancient biology to revolutionize modern medicine. This insight reframes how we think about health and cancer treatment by harnessing the immune system's innate power, moving beyond harsh treatments, and embracing nature’s original disease-fighter from within. The discovery is a call to awaken the body’s own god-given defense system for a future of empowered healing.

Camus

33,407 views • 8 months ago

We’re thrilled to share that our MERFISH+ preprint is now live on bioRxiv!👉 In this work, the Bintu and Zhu labs (UCSD) developed MERFISH+, a next-generation spatial genomics platform that combines genome-wide RNA and epigenetic imaging over a large field of view. By introducing acrydite-modified probes covalently anchored to hydrogels, MERFISH+ achieves remarkable imaging stability and enables >1,800-gene, multi-modal, and multi-month experiments. With this platform, they, together with the Chi lab at UCSD, profiled a whole developing human heart at 12 post-conception week with merely two slides, resulting in a total of 53 slides, 3.1 million single cells and more than 30 cell types. Building upon our previous 3D reconstruction and modeling framework, Spateo ( we reconstruct the 3D human heart that nicely captures the anatomical structure of the heart, including the intricate vasculature network. Sophisticated analyses provide a holistic view of an entire organ and enable systematic characterization of 3D cellular neighborhoods and transcriptional gradients of substructures such as the descending arteries. Furthermore, using a generative integration framework for spatial multimodal data (Spateo-VI), we harmonized these MERFISH+ transcriptomic and chromatin data to reconstruct a 3D spatially-resolved multi-omics atlas of the developing human heart, shared at and MERFISH+ thus sets a new standard for large-format, multi-omic spatial profiling, enabling holistic, 3D characterization of organs at subcellular resolution. Huge congratulations to first authors Colin Kern, qingquan Zhang, Yifan Lu , and Jacqueline Eschbach, and to all collaborators from the Bintu, Zhu, Chi, and Qiu labs for this amazing team effort. Thanks for your diligence, creativity, and hard work on this project. We’re grateful for support from Arc Institute and our generous donors. Our lab is expanding—if you’re excited about building the next generation of single-cell and spatial genomics techniques and predictive single cell and spatial foundation models, we’re hiring! If you are interested, please reach out to me via direct message or email at xiaojie@stanford.edu. We are excited for any potential collaborations along this line of research in Stanford, UCSF and Berkeley and other labs as well.

We’re thrilled to share that our MERFISH+ preprint is now live on bioRxiv!👉 In this work, the Bintu and Zhu labs (UCSD) developed MERFISH+, a next-generation spatial genomics platform that combines genome-wide RNA and epigenetic imaging over a large field of view. By introducing acrydite-modified probes covalently anchored to hydrogels, MERFISH+ achieves remarkable imaging stability and enables >1,800-gene, multi-modal, and multi-month experiments. With this platform, they, together with the Chi lab at UCSD, profiled a whole developing human heart at 12 post-conception week with merely two slides, resulting in a total of 53 slides, 3.1 million single cells and more than 30 cell types. Building upon our previous 3D reconstruction and modeling framework, Spateo ( we reconstruct the 3D human heart that nicely captures the anatomical structure of the heart, including the intricate vasculature network. Sophisticated analyses provide a holistic view of an entire organ and enable systematic characterization of 3D cellular neighborhoods and transcriptional gradients of substructures such as the descending arteries. Furthermore, using a generative integration framework for spatial multimodal data (Spateo-VI), we harmonized these MERFISH+ transcriptomic and chromatin data to reconstruct a 3D spatially-resolved multi-omics atlas of the developing human heart, shared at and MERFISH+ thus sets a new standard for large-format, multi-omic spatial profiling, enabling holistic, 3D characterization of organs at subcellular resolution. Huge congratulations to first authors Colin Kern, qingquan Zhang, Yifan Lu , and Jacqueline Eschbach, and to all collaborators from the Bintu, Zhu, Chi, and Qiu labs for this amazing team effort. Thanks for your diligence, creativity, and hard work on this project. We’re grateful for support from Arc Institute and our generous donors. Our lab is expanding—if you’re excited about building the next generation of single-cell and spatial genomics techniques and predictive single cell and spatial foundation models, we’re hiring! If you are interested, please reach out to me via direct message or email at [email protected]. We are excited for any potential collaborations along this line of research in Stanford, UCSF and Berkeley and other labs as well.

evo-devo

42,166 views • 8 months ago

New short course: Building Multimodal Search and RAG", by Weaviate AI Database's Sebastia(N_) Witalec ✊🏽✊🏾✊🏿. Contrastive learning is used to train models to map vectors into an embedding space by pulling similar concepts closer together and pushing dissimilar concepts away from each other. This technique is also used to train multimodal embedding models that capture semantic similarity across different modalities like text, images, and audio. These multimodal embeddings can be used to build multimodal search and RAG systems. In this course, you'll learn how contrastive learning works, and how to add multimodality to RAG – so your models can draw on diverse, relevant context to answer questions. For example, a query about a financial report might synthesize information from text snippets, graphs, tables, and slides. You will also learn how visual instruction tuning lets you integrate image understanding into language models, and build a multi-vector recommender system using Weaviate’s open-source vector database. Please sign up here:

New short course: Building Multimodal Search and RAG", by Weaviate AI Database's Sebastia(N_) Witalec ✊🏽✊🏾✊🏿. Contrastive learning is used to train models to map vectors into an embedding space by pulling similar concepts closer together and pushing dissimilar concepts away from each other. This technique is also used to train multimodal embedding models that capture semantic similarity across different modalities like text, images, and audio. These multimodal embeddings can be used to build multimodal search and RAG systems. In this course, you'll learn how contrastive learning works, and how to add multimodality to RAG – so your models can draw on diverse, relevant context to answer questions. For example, a query about a financial report might synthesize information from text snippets, graphs, tables, and slides. You will also learn how visual instruction tuning lets you integrate image understanding into language models, and build a multi-vector recommender system using Weaviate’s open-source vector database. Please sign up here:

Andrew Ng

104,371 views • 2 years ago

Explore state-of-the-art multimodal prompting in our new short course Large Multimodal Model Prompting with Gemini, taught by Erwin Huizenga in collaboration with Google Cloud. One interesting insight from this course: with multimodal models, prompt structure matters significantly. Placing text inputs, such as a patient's medical history, before image inputs, like an X-ray, can enhance the model's ability to contextualize and interpret visual data effectively. In other contexts, such as image captioning, you may get better results by putting the image first. Multimodal models behave differently than text-only LLMs, and effective prompting for models varies depending on the model you’re using. In this course you’ll learn how to effectively prompt Gemini models. Gemini's multimodal capabilities also enable new approaches in AI application development, for example: - The Gemini library handles various video formats (MP4, MOV, MPEG), streamlining applications using these formats. - Large context window (up to 1 million tokens) enables processing of extensive content, like analyzing multiple 50-minute videos simultaneously. - Function calling feature integrates real-time data (e.g., current exchange rates) into model responses. The course demonstrates building multimodal applications with real-world examples including document analyzers that reason across text and graphs simultaneously, video content extractors that find and timestamp specific information from multiple hours of footage, and automated expense report systems processing receipt images while cross-referencing company policies. Sign up here:

Explore state-of-the-art multimodal prompting in our new short course Large Multimodal Model Prompting with Gemini, taught by Erwin Huizenga in collaboration with Google Cloud. One interesting insight from this course: with multimodal models, prompt structure matters significantly. Placing text inputs, such as a patient's medical history, before image inputs, like an X-ray, can enhance the model's ability to contextualize and interpret visual data effectively. In other contexts, such as image captioning, you may get better results by putting the image first. Multimodal models behave differently than text-only LLMs, and effective prompting for models varies depending on the model you’re using. In this course you’ll learn how to effectively prompt Gemini models. Gemini's multimodal capabilities also enable new approaches in AI application development, for example: - The Gemini library handles various video formats (MP4, MOV, MPEG), streamlining applications using these formats. - Large context window (up to 1 million tokens) enables processing of extensive content, like analyzing multiple 50-minute videos simultaneously. - Function calling feature integrates real-time data (e.g., current exchange rates) into model responses. The course demonstrates building multimodal applications with real-world examples including document analyzers that reason across text and graphs simultaneously, video content extractors that find and timestamp specific information from multiple hours of footage, and automated expense report systems processing receipt images while cross-referencing company policies. Sign up here:

Andrew Ng

74,060 views • 1 year ago

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.

Google presents Still-Moving Customized Video Generation without Customized Video Data Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a text-to-image (T2I) model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth or StyleDrop). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.

AK

40,474 views • 2 years ago

The Bitter Lesson for Biology — Adam Green on Scaling Laws for Virtual Cells In this episode, the founder of Markov Biosciences, Adam Green, explains the "bitter lesson" for biology, the idea borrowed from Richard Sutton that large unbiased datasets and the right training objective tend to outcompete models with hard-coded rules and human priors. We talk a lot about virtual cells, sources of bias in data, and his evidence that virtual cells pre-trained on plain observational data show clean scaling laws, getting monotonically better at predicting unseen perturbations as the models grow, and beating a state-of-the-art model built specifically for that task. Our interview was about 3 hours long, but cut to 1.5 hours. My main goal was to understand Adam's perspective on virtual cells and his goal to "solve biology." Search for "The New Biology" on YouTube, Spotify, and Apple Podcasts. Timestamps: 00:00 — Cold open 01:58 — Clinical predictions from a virtual cell 05:38 — What is a "virtual cell"? 08:01 — The problems with single-cell RNA-seq 11:31 — The urns analogy 19:54 — Observational vs. perturbation 23:29 — The bitter lesson for biology 29:06 — Geometric Plackett-Luce 38:27 — Ablations and loss function 47:23 — Cells as specimens 59:26 — Antibody-Drug conjugates 1:11:16 — Will we ever understand the cell fully?

The Bitter Lesson for Biology — Adam Green on Scaling Laws for Virtual Cells In this episode, the founder of Markov Biosciences, Adam Green, explains the "bitter lesson" for biology, the idea borrowed from Richard Sutton that large unbiased datasets and the right training objective tend to outcompete models with hard-coded rules and human priors. We talk a lot about virtual cells, sources of bias in data, and his evidence that virtual cells pre-trained on plain observational data show clean scaling laws, getting monotonically better at predicting unseen perturbations as the models grow, and beating a state-of-the-art model built specifically for that task. Our interview was about 3 hours long, but cut to 1.5 hours. My main goal was to understand Adam's perspective on virtual cells and his goal to "solve biology." Search for "The New Biology" on YouTube, Spotify, and Apple Podcasts. Timestamps: 00:00 — Cold open 01:58 — Clinical predictions from a virtual cell 05:38 — What is a "virtual cell"? 08:01 — The problems with single-cell RNA-seq 11:31 — The urns analogy 19:54 — Observational vs. perturbation 23:29 — The bitter lesson for biology 29:06 — Geometric Plackett-Luce 38:27 — Ablations and loss function 47:23 — Cells as specimens 59:26 — Antibody-Drug conjugates 1:11:16 — Will we ever understand the cell fully?

Niko McCarty.

27,212 views • 1 month ago

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,657 views • 3 years ago

Do Vision-Language Models represent space, and how? Spatial terms like "left" or "right" may not be enough to match images with spatial descriptions, as we often overlook the different frames of reference (FoR) used by speakers and listeners. See Figure 1 for examples! Introducing the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to assess the spatial reasoning capabilities of VLMs. COMFORT includes systematically designed datasets and metrics that evaluate model performance, and their deeper linguistic competence, specifically the spatial knowledge encoded in their internal representations. Find out more in the video teaser! Almost all VLMs prefer the egocentric relative FoR with reflected transform, similar to English. Yet, we reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. A shortened version will appear in Pluralistic Alignment Workshop Pluralistic Alignment Workshop #NeurIPS2024. It seems that the ArXiv moderators put it on hold and are eager to give it a thorough read first🤣! So here is the Paper/Code/Data: This collaboration turns out to be amazing, jointly led by Brian Zheyuan Zhang, @Hu_FY_ Jayjun Lee, with so many contributions and insights from Freda Shi, Parisa Kordjamshidi Michigan SLED Lab. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning!

Do Vision-Language Models represent space, and how? Spatial terms like "left" or "right" may not be enough to match images with spatial descriptions, as we often overlook the different frames of reference (FoR) used by speakers and listeners. See Figure 1 for examples! Introducing the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to assess the spatial reasoning capabilities of VLMs. COMFORT includes systematically designed datasets and metrics that evaluate model performance, and their deeper linguistic competence, specifically the spatial knowledge encoded in their internal representations. Find out more in the video teaser! Almost all VLMs prefer the egocentric relative FoR with reflected transform, similar to English. Yet, we reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. A shortened version will appear in Pluralistic Alignment Workshop Pluralistic Alignment Workshop #NeurIPS2024. It seems that the ArXiv moderators put it on hold and are eager to give it a thorough read first🤣! So here is the Paper/Code/Data: This collaboration turns out to be amazing, jointly led by Brian Zheyuan Zhang, @Hu_FY_ Jayjun Lee, with so many contributions and insights from Freda Shi, Parisa Kordjamshidi Michigan SLED Lab. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning!

Martin Ziqiao Ma

35,565 views • 1 year ago

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models paper page: Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. In our experiments, JARVIS-1 exhibits nearly perfect performances across over 200 varying tasks from the Minecraft Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task. This represents a significant increase up to 5 times compared to previous records. Furthermore, we show that JARVIS-1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy.

AK

141,425 views • 2 years ago