Загрузка видео...

Не удалось загрузить видео

На главную

PantheonOS allows any biologist to perform complex data analyses of emerging single cell, multi-omics and spatial transcriptomics datasets end to end through AI agent and human collaboration. We are releasing six replays of use cases trajectories. Each "trajectory" is a complete end-to-end run, from prompt to analysis, to figures,...

Xiaojie_Qiu's profile picture

evo-devo

5,273 subscribers

13,429 просмотров • 4 месяцев назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 2

Фото профиля evo-devo
evo-devo4 месяцев назад

🤗

Фото профиля evo-devo
evo-devo4 месяцев назад

link to the replays:

Похожие видео

We are thrilled to share that our first paper from my new lab, Spateo ( for spatiotemporal modeling of molecular holograms, is now online in Cell: Spateo is a comprehensive analytical framework for 3D whole-embryo spatiotemporal modeling. Its advanced features include: • 3D alignment and reconstruction at the whole-mouse-embryo scale (see the animation). • 3D spatial domain digitization and cell-cell communication analysis to understand spatial gene expression gradients and both inter- and intracellular communication. • 3D morphometric and volumetric analyses along with 3D morphogenesis vector field modeling to quantify dynamics such as surface area, volume, and cell density across organs, and to dissect the interplay between morphogenesis factors and cell migration. • A “Google Earth”-like browser, Spateo-viewer ( and for interactive and intuitive exploration of 3D spatial data. • Additional features, such as RNA signal-based single-cell segmentation. We are also honored that Nature “News and Views” has highlighted this work as well: This is really an amazing outcome after two years' heroic revision process that rewrite the entire paper using a new data ( for whole mouse embryos.
1:24

We are thrilled to share that our first paper from my new lab, Spateo ( for spatiotemporal modeling of molecular holograms, is now online in Cell: Spateo is a comprehensive analytical framework for 3D whole-embryo spatiotemporal modeling. Its advanced features include: • 3D alignment and reconstruction at the whole-mouse-embryo scale (see the animation). • 3D spatial domain digitization and cell-cell communication analysis to understand spatial gene expression gradients and both inter- and intracellular communication. • 3D morphometric and volumetric analyses along with 3D morphogenesis vector field modeling to quantify dynamics such as surface area, volume, and cell density across organs, and to dissect the interplay between morphogenesis factors and cell migration. • A “Google Earth”-like browser, Spateo-viewer ( and for interactive and intuitive exploration of 3D spatial data. • Additional features, such as RNA signal-based single-cell segmentation. We are also honored that Nature “News and Views” has highlighted this work as well: This is really an amazing outcome after two years' heroic revision process that rewrite the entire paper using a new data ( for whole mouse embryos.

evo-devo

58,480 просмотров • 1 год назад

We’re thrilled to share that our MERFISH+ preprint is now live on bioRxiv!👉 In this work, the Bintu and Zhu labs (UCSD) developed MERFISH+, a next-generation spatial genomics platform that combines genome-wide RNA and epigenetic imaging over a large field of view. By introducing acrydite-modified probes covalently anchored to hydrogels, MERFISH+ achieves remarkable imaging stability and enables >1,800-gene, multi-modal, and multi-month experiments. With this platform, they, together with the Chi lab at UCSD, profiled a whole developing human heart at 12 post-conception week with merely two slides, resulting in a total of 53 slides, 3.1 million single cells and more than 30 cell types. Building upon our previous 3D reconstruction and modeling framework, Spateo ( we reconstruct the 3D human heart that nicely captures the anatomical structure of the heart, including the intricate vasculature network. Sophisticated analyses provide a holistic view of an entire organ and enable systematic characterization of 3D cellular neighborhoods and transcriptional gradients of substructures such as the descending arteries. Furthermore, using a generative integration framework for spatial multimodal data (Spateo-VI), we harmonized these MERFISH+ transcriptomic and chromatin data to reconstruct a 3D spatially-resolved multi-omics atlas of the developing human heart, shared at and MERFISH+ thus sets a new standard for large-format, multi-omic spatial profiling, enabling holistic, 3D characterization of organs at subcellular resolution. Huge congratulations to first authors Colin Kern, qingquan Zhang, Yifan Lu , and Jacqueline Eschbach, and to all collaborators from the Bintu, Zhu, Chi, and Qiu labs for this amazing team effort. Thanks for your diligence, creativity, and hard work on this project. We’re grateful for support from Arc Institute and our generous donors. Our lab is expanding—if you’re excited about building the next generation of single-cell and spatial genomics techniques and predictive single cell and spatial foundation models, we’re hiring! If you are interested, please reach out to me via direct message or email at xiaojie@stanford.edu. We are excited for any potential collaborations along this line of research in Stanford, UCSF and Berkeley and other labs as well.
0:42

We’re thrilled to share that our MERFISH+ preprint is now live on bioRxiv!👉 In this work, the Bintu and Zhu labs (UCSD) developed MERFISH+, a next-generation spatial genomics platform that combines genome-wide RNA and epigenetic imaging over a large field of view. By introducing acrydite-modified probes covalently anchored to hydrogels, MERFISH+ achieves remarkable imaging stability and enables >1,800-gene, multi-modal, and multi-month experiments. With this platform, they, together with the Chi lab at UCSD, profiled a whole developing human heart at 12 post-conception week with merely two slides, resulting in a total of 53 slides, 3.1 million single cells and more than 30 cell types. Building upon our previous 3D reconstruction and modeling framework, Spateo ( we reconstruct the 3D human heart that nicely captures the anatomical structure of the heart, including the intricate vasculature network. Sophisticated analyses provide a holistic view of an entire organ and enable systematic characterization of 3D cellular neighborhoods and transcriptional gradients of substructures such as the descending arteries. Furthermore, using a generative integration framework for spatial multimodal data (Spateo-VI), we harmonized these MERFISH+ transcriptomic and chromatin data to reconstruct a 3D spatially-resolved multi-omics atlas of the developing human heart, shared at and MERFISH+ thus sets a new standard for large-format, multi-omic spatial profiling, enabling holistic, 3D characterization of organs at subcellular resolution. Huge congratulations to first authors Colin Kern, qingquan Zhang, Yifan Lu , and Jacqueline Eschbach, and to all collaborators from the Bintu, Zhu, Chi, and Qiu labs for this amazing team effort. Thanks for your diligence, creativity, and hard work on this project. We’re grateful for support from Arc Institute and our generous donors. Our lab is expanding—if you’re excited about building the next generation of single-cell and spatial genomics techniques and predictive single cell and spatial foundation models, we’re hiring! If you are interested, please reach out to me via direct message or email at [email protected]. We are excited for any potential collaborations along this line of research in Stanford, UCSF and Berkeley and other labs as well.

evo-devo

42,315 просмотров • 10 месяцев назад

Excited to share our new work on building a multimodal atlas of human skin in health and inflammatory disease — a project I’m especially proud of, bringing together AI, high-throughput genomics, and clinical science to accelerate discovery. Over the past decade, single-cell genomics has transformed how we map cells in human tissues. But a major challenge remains: can we systematically decode how cells organize into functional niches in situ — including those invisible to standard histopathology? To address this, we integrated large-scale scRNA-seq, spatial transcriptomics, histopathology, and AI-driven modeling frameworks to build an in situ atlas of human skin across health and disease. Led by Lloyd Steele, an MD/PhD student working between Haniffa Lab and my lab at Wellcome Sanger Institute and Cambridge University . Another amazing collaboration with Muzz Haniffa, the mastermind behind the work as part of Human Cell Atlas. A key part of this study is that we didn’t build everything from scratch — we leveraged and combined AI methods that actually work! and showed how they can be used together to extract biological insight at scale. We used: • scArches to build and map into a reference scRNA-seq atlas of human skin: • NicheCompass to identify and characterize spatial niches: • MINT-Flow to extract microenvironment-induced cell states and gene programs: Together, these enabled an end-to-end workflow from atlas construction to spatial mapping, niche discovery, and cell state decoding. At scale, we integrated ~5 million cells and 100+ spatial sections, enabling a systematic view of tissue organization. Using this framework, we identified 26 niches in skin, including known histopathologic structures as well as hidden disease-associated niches not visible on H&E. Among the most striking findings were a resident memory T cell-rich sebaceous gland niche and a plasma cell-rich sweat gland niche, suggesting that appendageal structures act as active immunological microenvironments and may contribute to inflammatory memory and disease persistence. Importantly, this atlas is not just descriptive — it is usable. It can support mapping of new datasets, resolve finer cell types and niches, extract microenvironment-driven programs, and enable predictive analyses at scale. More broadly, this work shows what becomes possible when AI, spatial genomics, and atlas-scale data are integrated end-to-end: not just mapping tissues, but systematically decoding them. This was a massive collaboration, and I’m very grateful to the amazing scientists April Foster, Kenny Roberts, and Chloe Admane. Lloyd is an amazing scientist, and I’m especially excited for the community to see more of his work soon — stay tuned. The data and pre-trained models will be released soon. Preprint:
1:09

Excited to share our new work on building a multimodal atlas of human skin in health and inflammatory disease — a project I’m especially proud of, bringing together AI, high-throughput genomics, and clinical science to accelerate discovery. Over the past decade, single-cell genomics has transformed how we map cells in human tissues. But a major challenge remains: can we systematically decode how cells organize into functional niches in situ — including those invisible to standard histopathology? To address this, we integrated large-scale scRNA-seq, spatial transcriptomics, histopathology, and AI-driven modeling frameworks to build an in situ atlas of human skin across health and disease. Led by Lloyd Steele, an MD/PhD student working between Haniffa Lab and my lab at Wellcome Sanger Institute and Cambridge University . Another amazing collaboration with Muzz Haniffa, the mastermind behind the work as part of Human Cell Atlas. A key part of this study is that we didn’t build everything from scratch — we leveraged and combined AI methods that actually work! and showed how they can be used together to extract biological insight at scale. We used: • scArches to build and map into a reference scRNA-seq atlas of human skin: • NicheCompass to identify and characterize spatial niches: • MINT-Flow to extract microenvironment-induced cell states and gene programs: Together, these enabled an end-to-end workflow from atlas construction to spatial mapping, niche discovery, and cell state decoding. At scale, we integrated ~5 million cells and 100+ spatial sections, enabling a systematic view of tissue organization. Using this framework, we identified 26 niches in skin, including known histopathologic structures as well as hidden disease-associated niches not visible on H&E. Among the most striking findings were a resident memory T cell-rich sebaceous gland niche and a plasma cell-rich sweat gland niche, suggesting that appendageal structures act as active immunological microenvironments and may contribute to inflammatory memory and disease persistence. Importantly, this atlas is not just descriptive — it is usable. It can support mapping of new datasets, resolve finer cell types and niches, extract microenvironment-driven programs, and enable predictive analyses at scale. More broadly, this work shows what becomes possible when AI, spatial genomics, and atlas-scale data are integrated end-to-end: not just mapping tissues, but systematically decoding them. This was a massive collaboration, and I’m very grateful to the amazing scientists April Foster, Kenny Roberts, and Chloe Admane. Lloyd is an amazing scientist, and I’m especially excited for the community to see more of his work soon — stay tuned. The data and pre-trained models will be released soon. Preprint:

Mo Lotfollahi

11,832 просмотров • 5 месяцев назад

🔬 Exciting News! Our manuscript, "scGPT: toward building a foundation model for single-cell multi-omics using generative AI" is now finally published in Nature Methods (Nature Methods) 🎉 !!! (Re-)Introducing scGPT: A transformative foundation model engineered for single-cell omics analysis. Developed through the analysis of over 33 million human cells, scGPT sets a new benchmark for application versatility, offering both fine-tuning and zero-shot capabilities. Since its preprint in May 2023, scGPT has significantly impacted the field, evidenced by 13K+ installations, 600+ GitHub stars 🌟, and 40+ citations before its official publication! scGPT has been validated by numerous benchmark studies as a leading foundation model in single-cell analysis. Its pre-trained embeddings extend its utility beyond single-cell studies, enhancing a variety of downstream tasks including protein enrichment and genetic perturbation predictions. Some key updates lately: ---Expanded zero-shot applications for efficient reference mapping and integration, now with CellXGene census integration. ---Advanced perturbation analysis capabilities, including genome-scale perturb-seq data analysis and bulk sequencing data generalization. ---Upgraded scGPT package, offering versatile model loading compatible with PyTorch and flash-attn, for both GPU and CPU. ---Cloud-based scGPT applications for reference mapping, cell annotation, and gene regulatory network inference are available on ---Integration with Hugging Face for easier model training. Limitations: scGPT is an early foray into foundation models for single-cell omics, facing challenges like limited zero-shot learning in some tasks, pretraining constraints, data quality issues, and evaluation limitations. See our Supplementary Notes for details. 🚀 Future Work? Short-Term Goals: 1. Releasing a Mouse Model for broader analysis. 2. Developing a comprehensive evaluation suite for foundation models in single-cell analysis. 3. Creating a foundation model for single-cell spatial omics. 4. Enhancing zero-shot capacity by integrating scGPT with RAG (e.g., knowledge graphs). Long-Term Goals: 1. Expanding scGPT for comprehensive single-cell multi-omics analysis. 2. Developing an in-silico perturbation model for predicting genetic perturbation effects. 3. Merging scGPT with multi-modal genomic sequence models for a deeper understanding of cell biology. 📚 Access the paper on Nature Methods: 🔬Preprint in Bioarixv: 💻 All our codes/data/weights are open source: Wholehearted congratulations to all the authors, especially the two co-first authors, Haotian (Haotian Cui ) and Chloe (ChloeXWang), who are really the emerging superstars in AI and biology! Vector Institute Peter Munk Cardiac Centre AI U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology University Health Network University of Toronto #scGPT #GenerativeAI #AI4Science #Combio #opensource
0:52

🔬 Exciting News! Our manuscript, "scGPT: toward building a foundation model for single-cell multi-omics using generative AI" is now finally published in Nature Methods (Nature Methods) 🎉 !!! (Re-)Introducing scGPT: A transformative foundation model engineered for single-cell omics analysis. Developed through the analysis of over 33 million human cells, scGPT sets a new benchmark for application versatility, offering both fine-tuning and zero-shot capabilities. Since its preprint in May 2023, scGPT has significantly impacted the field, evidenced by 13K+ installations, 600+ GitHub stars 🌟, and 40+ citations before its official publication! scGPT has been validated by numerous benchmark studies as a leading foundation model in single-cell analysis. Its pre-trained embeddings extend its utility beyond single-cell studies, enhancing a variety of downstream tasks including protein enrichment and genetic perturbation predictions. Some key updates lately: ---Expanded zero-shot applications for efficient reference mapping and integration, now with CellXGene census integration. ---Advanced perturbation analysis capabilities, including genome-scale perturb-seq data analysis and bulk sequencing data generalization. ---Upgraded scGPT package, offering versatile model loading compatible with PyTorch and flash-attn, for both GPU and CPU. ---Cloud-based scGPT applications for reference mapping, cell annotation, and gene regulatory network inference are available on ---Integration with Hugging Face for easier model training. Limitations: scGPT is an early foray into foundation models for single-cell omics, facing challenges like limited zero-shot learning in some tasks, pretraining constraints, data quality issues, and evaluation limitations. See our Supplementary Notes for details. 🚀 Future Work? Short-Term Goals: 1. Releasing a Mouse Model for broader analysis. 2. Developing a comprehensive evaluation suite for foundation models in single-cell analysis. 3. Creating a foundation model for single-cell spatial omics. 4. Enhancing zero-shot capacity by integrating scGPT with RAG (e.g., knowledge graphs). Long-Term Goals: 1. Expanding scGPT for comprehensive single-cell multi-omics analysis. 2. Developing an in-silico perturbation model for predicting genetic perturbation effects. 3. Merging scGPT with multi-modal genomic sequence models for a deeper understanding of cell biology. 📚 Access the paper on Nature Methods: 🔬Preprint in Bioarixv: 💻 All our codes/data/weights are open source: Wholehearted congratulations to all the authors, especially the two co-first authors, Haotian (Haotian Cui ) and Chloe (ChloeXWang), who are really the emerging superstars in AI and biology! Vector Institute Peter Munk Cardiac Centre AI U of T Department of Computer Science Department of Laboratory Medicine & Pathobiology University Health Network University of Toronto #scGPT #GenerativeAI #AI4Science #Combio #opensource

Bo Wang

199,776 просмотров • 2 лет назад

A talk on verifiable environments for agents in biology 0:00 Where does experimental data come from? 1:41 Modern bio research organized around measurement 2:44 Data analysis is an executable substrate for science 3:23 Five years of learnings building products for pharma 4:13 Trying to use coding models for biology workflows 5:40 Why frontier models can't be trusted (yet) 6:44 Sequencing based spatial analysis 7:38 SpatialBench design principles 8:26 Anatomy of a single evaluation 9:40 Human verification and early long horizon extensions 13:12 Grading functions and the case against rubrics 14:07 New work in multi-omics, therapeutics, biosecurity
17:29

A talk on verifiable environments for agents in biology 0:00 Where does experimental data come from? 1:41 Modern bio research organized around measurement 2:44 Data analysis is an executable substrate for science 3:23 Five years of learnings building products for pharma 4:13 Trying to use coding models for biology workflows 5:40 Why frontier models can't be trusted (yet) 6:44 Sequencing based spatial analysis 7:38 SpatialBench design principles 8:26 Anatomy of a single evaluation 9:40 Human verification and early long horizon extensions 13:12 Grading functions and the case against rubrics 14:07 New work in multi-omics, therapeutics, biosecurity

Kenny Workman

92,323 просмотров • 1 месяц назад

🚀 Introducing scGPT-spatial! 🧬🌍 A game-changing spatial-omic foundation model, built on the powerful scGPT framework with MoE (mixture of experts) and continually pretrained on a massive 30 million spatial single-cell profiles! 🧠 What’s the challenge? Spatial transcriptomics is next-level complex—not only must we model single-cell/spot profiles, but we also need to capture intricate spatial relationships while handling diverse sequencing protocols (imaging-based vs. sequencing-based). 🔥 Why scGPT-spatial? ✨ A Spatial-omic Foundation Model with Continual Pretraining – Built on scGPT’s robust initialization, it unlocks spatial context in tissues. ✨ SpatialHuman30M Dataset – The largest curated dataset: 30M profiles from Visium, Visium HD, Xenium, and MERFISH across 821 slides. ✨ Revolutionary MoE Decoders – A cutting-edge Mixture of Experts (MoE) architecture for protocol-aware gene expression decoding. ✨ Spatially-Aware Training Strategy – A neighborhood-based masked reconstruction approach to capture complex cell-type colocalization. ✨ Multi-Modal & Multi-Slide Integration – Seamless clustering & spatial domain identification across slides and modalities. ✨ Cell-Type Deconvolution & Gene Imputation – Unlocks cross-resolution & cross-modality harmonization with fine-tuned embeddings. 📄 Read the preprint: 💻 Explore the code/weights: #SpatialTranscriptomics #SingleCell #AIResearch #MachineLearning #SpatialData Huge shoutout to the incredible PHD students Chloe (ChloeXWang) and Haotian (Haotian Cui) for leading this groundbreaking project! 🎉 Massive thanks to our amazing co-authors Andrew, Ronald, and Hani (Hani Goodarzi) from Arc Institute—this work wouldn't have been possible without you! 👏
0:37

🚀 Introducing scGPT-spatial! 🧬🌍 A game-changing spatial-omic foundation model, built on the powerful scGPT framework with MoE (mixture of experts) and continually pretrained on a massive 30 million spatial single-cell profiles! 🧠 What’s the challenge? Spatial transcriptomics is next-level complex—not only must we model single-cell/spot profiles, but we also need to capture intricate spatial relationships while handling diverse sequencing protocols (imaging-based vs. sequencing-based). 🔥 Why scGPT-spatial? ✨ A Spatial-omic Foundation Model with Continual Pretraining – Built on scGPT’s robust initialization, it unlocks spatial context in tissues. ✨ SpatialHuman30M Dataset – The largest curated dataset: 30M profiles from Visium, Visium HD, Xenium, and MERFISH across 821 slides. ✨ Revolutionary MoE Decoders – A cutting-edge Mixture of Experts (MoE) architecture for protocol-aware gene expression decoding. ✨ Spatially-Aware Training Strategy – A neighborhood-based masked reconstruction approach to capture complex cell-type colocalization. ✨ Multi-Modal & Multi-Slide Integration – Seamless clustering & spatial domain identification across slides and modalities. ✨ Cell-Type Deconvolution & Gene Imputation – Unlocks cross-resolution & cross-modality harmonization with fine-tuned embeddings. 📄 Read the preprint: 💻 Explore the code/weights: #SpatialTranscriptomics #SingleCell #AIResearch #MachineLearning #SpatialData Huge shoutout to the incredible PHD students Chloe (ChloeXWang) and Haotian (Haotian Cui) for leading this groundbreaking project! 🎉 Massive thanks to our amazing co-authors Andrew, Ronald, and Hani (Hani Goodarzi) from Arc Institute—this work wouldn't have been possible without you! 👏

Bo Wang

59,081 просмотров • 1 год назад

🚀 Introducing PantheonOS ( A Fully Open-Source Agent OS for Science PantheonOS began as a research project in my Stanford lab and has since evolved into a vision to redefine data science in the era of AI—starting with computational biology, especially single-cell and spatial genomics. PantheonOS is a general agent platform built from the ground up. It is arguably the first distributed agent framework designed for scientific data analysis. 🔑 Key Features 1. Multi-Agent Collaboration – Built-in paradigms for distributed, cross-machine cooperation among agents and toolsets. 2. Native Toolset Support – Python, R, Julia, LaTeX, and more—designed for real scientific workflows. 3. Modular & Extensible – Developer-friendly design with shallow wrappers, plus LLM-driven toolset generation. 4. Evolvable Agents – Capable of evolving large-scale code projects to achieve superhuman performance (e.g., evolving upon the original Harmony [I Korsunsky, 2019, Nature Biotechnology] and Scanorama [BL Hie, 2019, Nature Biotechnology] implementations), and even evolving the system itself to adapt to new fields. 🎉 Stepwise Release Strategy We’re releasing PantheonOS in stages: Pantheon-CLI (today!), followed by Pantheon-Lab, Pantheon-Notebook, Pantheon-Slack, and more. 🌟 Pantheon-CLI Highlights - We're not just building another CLI tool. We're defining how scientists will interact with data in the AI era. - Open, Powerful, Python-First – The first fully open-source, endlessly extendable scientific “vibe analysis” framework. - Mixed Programming Magic – Combine Python, natural language, R, or Julia—seamlessly in the same environment. - PhD-Level Assistant – A command-line agent for complex real-world genomics and beyond, handling workflows at the PhD level. - Privacy by Design – Run entirely offline with local LLMs—your data never leaves your computer. ✅ Proven Applications (10 Demonstrations) Computational biology: 1. ATAC-seq: From raw reads to peak matrix 2. RNA-seq: From raw reads to expression matrix 3. Complex single-cell workflows (PhD-level) 4. Hybrid natural language + R for Seurat annotation 5. Learning from web tutorials + invoking single-cell foundation models 6. Cell segmentation on 10x Genomics HD Visium data And beyond: 7. Mixed Python & R programming examples 8. Molecular docking & structural analysis 9. Exploratory factor analysis for behavioral survey data 10. Customer segmentation & finance analytics 🌐 Learn More & Get Started Website: Pantheon-CLI Documentation: GitHub Repo: 💬 Join our community: PantheonOS Slack: PantheonOS Discord:
1:16

🚀 Introducing PantheonOS ( A Fully Open-Source Agent OS for Science PantheonOS began as a research project in my Stanford lab and has since evolved into a vision to redefine data science in the era of AI—starting with computational biology, especially single-cell and spatial genomics. PantheonOS is a general agent platform built from the ground up. It is arguably the first distributed agent framework designed for scientific data analysis. 🔑 Key Features 1. Multi-Agent Collaboration – Built-in paradigms for distributed, cross-machine cooperation among agents and toolsets. 2. Native Toolset Support – Python, R, Julia, LaTeX, and more—designed for real scientific workflows. 3. Modular & Extensible – Developer-friendly design with shallow wrappers, plus LLM-driven toolset generation. 4. Evolvable Agents – Capable of evolving large-scale code projects to achieve superhuman performance (e.g., evolving upon the original Harmony [I Korsunsky, 2019, Nature Biotechnology] and Scanorama [BL Hie, 2019, Nature Biotechnology] implementations), and even evolving the system itself to adapt to new fields. 🎉 Stepwise Release Strategy We’re releasing PantheonOS in stages: Pantheon-CLI (today!), followed by Pantheon-Lab, Pantheon-Notebook, Pantheon-Slack, and more. 🌟 Pantheon-CLI Highlights - We're not just building another CLI tool. We're defining how scientists will interact with data in the AI era. - Open, Powerful, Python-First – The first fully open-source, endlessly extendable scientific “vibe analysis” framework. - Mixed Programming Magic – Combine Python, natural language, R, or Julia—seamlessly in the same environment. - PhD-Level Assistant – A command-line agent for complex real-world genomics and beyond, handling workflows at the PhD level. - Privacy by Design – Run entirely offline with local LLMs—your data never leaves your computer. ✅ Proven Applications (10 Demonstrations) Computational biology: 1. ATAC-seq: From raw reads to peak matrix 2. RNA-seq: From raw reads to expression matrix 3. Complex single-cell workflows (PhD-level) 4. Hybrid natural language + R for Seurat annotation 5. Learning from web tutorials + invoking single-cell foundation models 6. Cell segmentation on 10x Genomics HD Visium data And beyond: 7. Mixed Python & R programming examples 8. Molecular docking & structural analysis 9. Exploratory factor analysis for behavioral survey data 10. Customer segmentation & finance analytics 🌐 Learn More & Get Started Website: Pantheon-CLI Documentation: GitHub Repo: 💬 Join our community: PantheonOS Slack: PantheonOS Discord:

evo-devo

17,431 просмотров • 1 год назад

Today, we’re pushing a major update to Edison Analysis, our data analysis agent, which is tuned for scientific research and SOTA across data analysis benchmarks. In contrast to Kosmos, which runs for 6-12 hours and produces tens of thousands of lines of code, Edison Analysis runs for seconds to minutes and is best for specific, well-defined computational tasks. It is available both on our platform under the Analysis tab, and via API, and costs only one credit per run, so it is available to users on both free and paid tiers. Edison Analysis is a modified version of the data analysis agent Kosmos uses in its trajectories. Try it out! One of the most important improvements over our previous data analysis agents has been the addition of a specialized data retrieval tool. Edison Analysis can either use this tool to access data, or can pull data down directly via API. To evaluate this tool, we ranked the most commonly used public data repositories across recent papers from BioRxiv, and created a new benchmark that measures the ability of a language agent system to retrieve raw data from those sources. Edison Analysis gets 71% on this benchmark, and we’ll be working to increase this over time. You can read more about our benchmarks in the our blog post, link below. Some features worth highlighting: 1. Edison Analysis produces a report on the analysis it runs, along with a Jupyter notebook that you can download to reproduce the analysis yourself. Every figure it produces is linked back to the specific lines of code used to produce the figure, to make it easy to reproduce. 2. It works well with both Python and R. 3. One of the best uses for Edison Analysis is to use it to retrieve datasets that you can then analyze with Kosmos. We have a bunch of major improvements to Edison Analysis coming in the next few months that we’re excited to share. In the meantime, congratulations to the team, especially Ludovico Mitchener, Jon Laurent, Conor Igoe , Alex Andonian, and many more.
1:12

Today, we’re pushing a major update to Edison Analysis, our data analysis agent, which is tuned for scientific research and SOTA across data analysis benchmarks. In contrast to Kosmos, which runs for 6-12 hours and produces tens of thousands of lines of code, Edison Analysis runs for seconds to minutes and is best for specific, well-defined computational tasks. It is available both on our platform under the Analysis tab, and via API, and costs only one credit per run, so it is available to users on both free and paid tiers. Edison Analysis is a modified version of the data analysis agent Kosmos uses in its trajectories. Try it out! One of the most important improvements over our previous data analysis agents has been the addition of a specialized data retrieval tool. Edison Analysis can either use this tool to access data, or can pull data down directly via API. To evaluate this tool, we ranked the most commonly used public data repositories across recent papers from BioRxiv, and created a new benchmark that measures the ability of a language agent system to retrieve raw data from those sources. Edison Analysis gets 71% on this benchmark, and we’ll be working to increase this over time. You can read more about our benchmarks in the our blog post, link below. Some features worth highlighting: 1. Edison Analysis produces a report on the analysis it runs, along with a Jupyter notebook that you can download to reproduce the analysis yourself. Every figure it produces is linked back to the specific lines of code used to produce the figure, to make it easy to reproduce. 2. It works well with both Python and R. 3. One of the best uses for Edison Analysis is to use it to retrieve datasets that you can then analyze with Kosmos. We have a bunch of major improvements to Edison Analysis coming in the next few months that we’re excited to share. In the meantime, congratulations to the team, especially Ludovico Mitchener, Jon Laurent, Conor Igoe , Alex Andonian, and many more.

Sam Rodriques

62,001 просмотров • 9 месяцев назад

🚀 We are hiring! 𝗣𝗵𝗗 𝗦𝘁𝘂𝗱𝗲𝗻𝘁 𝗶𝗻 𝗖𝗔𝗥-𝗧 𝗖𝗲𝗹𝗹𝘂𝗹𝗮𝗿 𝗧𝗵𝗲𝗿𝗮𝗽𝘆 𝗮𝗻𝗱 𝗖𝗮𝗻𝗰𝗲𝗿 𝗡𝗲𝘂𝗿𝗼𝗶𝗺𝗺𝘂𝗻𝗼𝗹𝗼𝗴𝘆 We are looking for a passionate and motivated PhD student to join an exciting project focused on how nerve, immune and cancer cell interactions shape metastatic tumor behavior and resistance to cellular therapies, especially CAR-T cells. The project combines: • Whole-body tissue clearing • Light-sheet microscopy • wildDISCO-based immunolabeling • CAR-T and immune cell tracking in metastatic cancer • Spatial proteomics and 3D spatial biology • AI-based analysis of large-scale imaging datasets This is a great opportunity for someone with a background in cancer biology, immunology, neuroscience, biomedical sciences, biomedical engineering or a related field who wants to work at the interface of experimental biology, advanced imaging and AI. Apply here: Please share with excellent candidates who may be interested. #Hiring #PhDPosition #CARTcells #CellularTherapy #CancerResearch #Neuroimmunology #Immunotherapy #Metastasis #SpatialBiology #AIinBiology #Microscopy #LMUMunich #HelmholtzMunich #ErturkLab
0:49

🚀 We are hiring! 𝗣𝗵𝗗 𝗦𝘁𝘂𝗱𝗲𝗻𝘁 𝗶𝗻 𝗖𝗔𝗥-𝗧 𝗖𝗲𝗹𝗹𝘂𝗹𝗮𝗿 𝗧𝗵𝗲𝗿𝗮𝗽𝘆 𝗮𝗻𝗱 𝗖𝗮𝗻𝗰𝗲𝗿 𝗡𝗲𝘂𝗿𝗼𝗶𝗺𝗺𝘂𝗻𝗼𝗹𝗼𝗴𝘆 We are looking for a passionate and motivated PhD student to join an exciting project focused on how nerve, immune and cancer cell interactions shape metastatic tumor behavior and resistance to cellular therapies, especially CAR-T cells. The project combines: • Whole-body tissue clearing • Light-sheet microscopy • wildDISCO-based immunolabeling • CAR-T and immune cell tracking in metastatic cancer • Spatial proteomics and 3D spatial biology • AI-based analysis of large-scale imaging datasets This is a great opportunity for someone with a background in cancer biology, immunology, neuroscience, biomedical sciences, biomedical engineering or a related field who wants to work at the interface of experimental biology, advanced imaging and AI. Apply here: Please share with excellent candidates who may be interested. #Hiring #PhDPosition #CARTcells #CellularTherapy #CancerResearch #Neuroimmunology #Immunotherapy #Metastasis #SpatialBiology #AIinBiology #Microscopy #LMUMunich #HelmholtzMunich #ErturkLab

Ali Max Erturk

14,771 просмотров • 4 месяцев назад

Excited to share TERRA, a tissue world model 🧬 Over ~1.5 years we ran a large data-generation + modelling effort to build a world model for human tissues, pretrained on 112M cells from spatial transcriptomics (mostly Xenium 5000-plex + public data). It's built on one of the largest human spatial transcriptomics corpora assembled to date, spanning 20 tissues across development, health and 26 disease conditions, ~two-thirds newly generated in-house. Why a "world model" for tissue? Images have universal representations (ViT/DINOv3), so do proteins (ESM, Alex Rives) and pathology (UNI, Faisal Mahmood). We've worked hard to build something similar for human tissue: one model that captures its multi-scale logic, genes → cells → their native microenvironments. Like the JEPA approach Yann LeCun has championed, TERRA learns by prediction in embedding space, but for human tissue. How it works: it tokenises each cell together with its nearest neighbours into one sequence while keeping every gene's identity, then masks part of a neighbourhood and predicts the representation of the hidden part, not raw noisy counts. From one backbone it reads out three scales, gene embeddings (what a gene is doing in a cell and its niche), cell embeddings (cell type and state) and neighbourhood embeddings (the niche), and because it keeps gene-level resolution it can knock a gene out in silico and predict the response. Applied entirely zero-shot, TERRA maps and perturbs human tissue across unseen organs, diseases and technologies, outperforming existing spatial approaches. Three take-homes: 1️⃣ One model, any tissue. A single pretrained backbone provides tissue representations zero-shot, handling genes, cells and niches across organs and platforms, off the shelf. 2️⃣ New biology, development to clinic. We built a new spatial atlas of the developing human pancreas and found an islet-associated capillary state that looks like a precursor of mature islet vasculature. In kidney, TERRA's in silico knockouts predicted the tissue-injury programme from cancer immunotherapy (checkpoint blockade), confirmed in treated kidneys, detected in blood, and linked to declining kidney function. 3️⃣ A grammar of tissue architecture. By coupling each cell's state to its niche, TERRA defines recurring cross-organ "archetypes" of macrophage neighbourhoods, including a tumour-boundary niche that tracks poor survival in kidney cancer. TERRA is already in use: it powered our recent skin atlas of hidden immune-memory niches ( with more studies coming soon. This was an amazing collaboration between clinicians, machine-learning scientists and cell biologists 🙏 Led by Sebastian Birk, Vali Sanian Amirhosein Vahidi, Samuel Ogden, Daniyal Jafree, Adib Miraki, Carlo Leonardi and Arpit Merchant, with Lassi Paavolainen, Menna Clatworthy, Omer Ali Bayraktar, Muzlifah Haniffa, Tom Mitchell and Mostafa Bakhti. Huge thanks too to everyone who shared data and helped along the way. What excites me most is seeing how the community builds on this. The model, code and tutorials are all public, so anyone can run TERRA on their own tissues, extend it, or build new models on top. Huge thanks to the whole team across Wellcome Sanger Institute and our many collaborators. 📄 Paper: 💻 Code: 🤗 Model: #SpatialTranscriptomics #SpatialGenomics #FoundationModels #AI4Science #MachineLearning #ComputationalBiology #SingleCell #WorldModels
2:15

Excited to share TERRA, a tissue world model 🧬 Over ~1.5 years we ran a large data-generation + modelling effort to build a world model for human tissues, pretrained on 112M cells from spatial transcriptomics (mostly Xenium 5000-plex + public data). It's built on one of the largest human spatial transcriptomics corpora assembled to date, spanning 20 tissues across development, health and 26 disease conditions, ~two-thirds newly generated in-house. Why a "world model" for tissue? Images have universal representations (ViT/DINOv3), so do proteins (ESM, Alex Rives) and pathology (UNI, Faisal Mahmood). We've worked hard to build something similar for human tissue: one model that captures its multi-scale logic, genes → cells → their native microenvironments. Like the JEPA approach Yann LeCun has championed, TERRA learns by prediction in embedding space, but for human tissue. How it works: it tokenises each cell together with its nearest neighbours into one sequence while keeping every gene's identity, then masks part of a neighbourhood and predicts the representation of the hidden part, not raw noisy counts. From one backbone it reads out three scales, gene embeddings (what a gene is doing in a cell and its niche), cell embeddings (cell type and state) and neighbourhood embeddings (the niche), and because it keeps gene-level resolution it can knock a gene out in silico and predict the response. Applied entirely zero-shot, TERRA maps and perturbs human tissue across unseen organs, diseases and technologies, outperforming existing spatial approaches. Three take-homes: 1️⃣ One model, any tissue. A single pretrained backbone provides tissue representations zero-shot, handling genes, cells and niches across organs and platforms, off the shelf. 2️⃣ New biology, development to clinic. We built a new spatial atlas of the developing human pancreas and found an islet-associated capillary state that looks like a precursor of mature islet vasculature. In kidney, TERRA's in silico knockouts predicted the tissue-injury programme from cancer immunotherapy (checkpoint blockade), confirmed in treated kidneys, detected in blood, and linked to declining kidney function. 3️⃣ A grammar of tissue architecture. By coupling each cell's state to its niche, TERRA defines recurring cross-organ "archetypes" of macrophage neighbourhoods, including a tumour-boundary niche that tracks poor survival in kidney cancer. TERRA is already in use: it powered our recent skin atlas of hidden immune-memory niches ( with more studies coming soon. This was an amazing collaboration between clinicians, machine-learning scientists and cell biologists 🙏 Led by Sebastian Birk, Vali Sanian Amirhosein Vahidi, Samuel Ogden, Daniyal Jafree, Adib Miraki, Carlo Leonardi and Arpit Merchant, with Lassi Paavolainen, Menna Clatworthy, Omer Ali Bayraktar, Muzlifah Haniffa, Tom Mitchell and Mostafa Bakhti. Huge thanks too to everyone who shared data and helped along the way. What excites me most is seeing how the community builds on this. The model, code and tutorials are all public, so anyone can run TERRA on their own tissues, extend it, or build new models on top. Huge thanks to the whole team across Wellcome Sanger Institute and our many collaborators. 📄 Paper: 💻 Code: 🤗 Model: #SpatialTranscriptomics #SpatialGenomics #FoundationModels #AI4Science #MachineLearning #ComputationalBiology #SingleCell #WorldModels

Mo Lotfollahi

50,993 просмотров • 1 месяц назад

Exciting update on PantheonOS: Introducing Pantheon-Notebook & Pantheon-CLI — the first fully open-source, Python-based agentic tools that go beyond Claude Code in the field of data analysis. Pantheon-CLI runs entirely on your computer or server, supports 60+ tools and 50+ databases, and can call any Python, R, or Julia package alongside natural language. Chat with your data directly. It look like python-claude-code, but more appreciate for data analysis. Pantheon-Notebook brings the same agentic framework into Jupyter! Not just for writing code, it can also run and revise code automatically to generate the correct result, and even operate on files and study from website — beyond what any other tool can do! With Pantheon, you mix natural language + programming in one workflow, focusing on discovery instead of syntax barriers. We've applied Pantheon in some real-world cases: finance (customer explore), biology (Seurat, cell segmentation, annotation), sociology (survey analysis), and drug discovery (molecular docking). Pantheon is not just a CLI or a plugin — it's an agentic operating system for science, spanning both terminal and notebook. Why not try it now? We are actively preparing publications from this series of projects. Major contributors will be recognized in our GitHub repository and listed as key authors in these manuscripts. Feel free to reach out for collaborations, research assistant positions, visiting opportunities, rotation project or future PhD projects.
1:16

Exciting update on PantheonOS: Introducing Pantheon-Notebook & Pantheon-CLI — the first fully open-source, Python-based agentic tools that go beyond Claude Code in the field of data analysis. Pantheon-CLI runs entirely on your computer or server, supports 60+ tools and 50+ databases, and can call any Python, R, or Julia package alongside natural language. Chat with your data directly. It look like python-claude-code, but more appreciate for data analysis. Pantheon-Notebook brings the same agentic framework into Jupyter! Not just for writing code, it can also run and revise code automatically to generate the correct result, and even operate on files and study from website — beyond what any other tool can do! With Pantheon, you mix natural language + programming in one workflow, focusing on discovery instead of syntax barriers. We've applied Pantheon in some real-world cases: finance (customer explore), biology (Seurat, cell segmentation, annotation), sociology (survey analysis), and drug discovery (molecular docking). Pantheon is not just a CLI or a plugin — it's an agentic operating system for science, spanning both terminal and notebook. Why not try it now? We are actively preparing publications from this series of projects. Major contributors will be recognized in our GitHub repository and listed as key authors in these manuscripts. Feel free to reach out for collaborations, research assistant positions, visiting opportunities, rotation project or future PhD projects.

evo-devo

45,652 просмотров • 1 год назад

⚡🎉 We are thrilled to introduce VORTEX, an AI-powered computational framework for predicting 3D Spatial Transcriptomics (ST) using 3D tissue images and minimal 2D ST! 🧬 By combining cutting-edge 3D non-destructive tissue imaging with AI, VORTEX imputes the 3D molecular landscape of large tissue samples in a cost-effective and scalable manner. 🧠💡Our approach: By pretraining on diverse 3D morphology–2D transcriptomic pairs from heterogeneous tissue samples, and then fine-tuning on minimal 2D ST data from a volume of interest, VORTEX leverages both generic tissue-specific and sample-specific morphomolecular correlates to predict 3D ST. Congratulations to our superstar co-leads Cristina Almagro Pérez and Andrew H. Song, this was an exciting collaboration with Jonathan Liu Sizun Jiang Ali Bashashati. Preprint: Demo: Read the excellent blog from our superstar grad student Cristina Almagro Pérez: Also see our previous work on 3D Computational Pathology from Andrew H. Song published in Cell last year: Stay tuned for more to come.
0:51

⚡🎉 We are thrilled to introduce VORTEX, an AI-powered computational framework for predicting 3D Spatial Transcriptomics (ST) using 3D tissue images and minimal 2D ST! 🧬 By combining cutting-edge 3D non-destructive tissue imaging with AI, VORTEX imputes the 3D molecular landscape of large tissue samples in a cost-effective and scalable manner. 🧠💡Our approach: By pretraining on diverse 3D morphology–2D transcriptomic pairs from heterogeneous tissue samples, and then fine-tuning on minimal 2D ST data from a volume of interest, VORTEX leverages both generic tissue-specific and sample-specific morphomolecular correlates to predict 3D ST. Congratulations to our superstar co-leads Cristina Almagro Pérez and Andrew H. Song, this was an exciting collaboration with Jonathan Liu Sizun Jiang Ali Bashashati. Preprint: Demo: Read the excellent blog from our superstar grad student Cristina Almagro Pérez: Also see our previous work on 3D Computational Pathology from Andrew H. Song published in Cell last year: Stay tuned for more to come.

Faisal Mahmood

18,047 просмотров • 1 год назад

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.
1:29

Alibaba presents MIMO Controllable Character Video Synthesis with Spatial Decomposed Modeling Character video synthesis aims to produce realistic videos of animatable characters within lifelike scenes. As a fundamental problem in the computer vision and graphics community, 3D works typically require multi-view captures for per-case training, which severely limits their applicability of modeling arbitrary characters in a short time. Recent 2D methods break this limitation via pre-trained diffusion models, but they struggle for pose generality and scene interaction. To this end, we propose MIMO, a novel framework which can not only synthesize character videos with controllable attributes (i.e., character, motion and scene) provided by simple user inputs, but also simultaneously achieve advanced scalability to arbitrary characters, generality to novel 3D motions, and applicability to interactive real-world scenes in a unified framework. The core idea is to encode the 2D video to compact spatial codes, considering the inherent 3D nature of video occurrence. Concretely, we lift the 2D frame pixels into 3D using monocular depth estimators, and decompose the video clip to three spatial components (i.e., main human, underlying scene, and floating occlusion) in hierarchical layers based on the 3D depth. These components are further encoded to canonical identity code, structured motion code and full scene code, which are utilized as control signals of synthesis process. The design of spatial decomposed modeling enables flexible user control, complex motion expression, as well as 3D-aware synthesis for scene interactions. Experimental results demonstrate effectiveness and robustness of the proposed method.

AK

149,079 просмотров • 1 год назад

Do Vision-Language Models represent space, and how? Spatial terms like "left" or "right" may not be enough to match images with spatial descriptions, as we often overlook the different frames of reference (FoR) used by speakers and listeners. See Figure 1 for examples! Introducing the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to assess the spatial reasoning capabilities of VLMs. COMFORT includes systematically designed datasets and metrics that evaluate model performance, and their deeper linguistic competence, specifically the spatial knowledge encoded in their internal representations. Find out more in the video teaser! Almost all VLMs prefer the egocentric relative FoR with reflected transform, similar to English. Yet, we reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. A shortened version will appear in Pluralistic Alignment Workshop Pluralistic Alignment Workshop #NeurIPS2024. It seems that the ArXiv moderators put it on hold and are eager to give it a thorough read first🤣! So here is the Paper/Code/Data: This collaboration turns out to be amazing, jointly led by Brian Zheyuan Zhang, @Hu_FY_ Jayjun Lee, with so many contributions and insights from Freda Shi, Parisa Kordjamshidi Michigan SLED Lab. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning!
0:30

Do Vision-Language Models represent space, and how? Spatial terms like "left" or "right" may not be enough to match images with spatial descriptions, as we often overlook the different frames of reference (FoR) used by speakers and listeners. See Figure 1 for examples! Introducing the COnsistent Multilingual Frame Of Reference Test (COMFORT), an evaluation protocol to assess the spatial reasoning capabilities of VLMs. COMFORT includes systematically designed datasets and metrics that evaluate model performance, and their deeper linguistic competence, specifically the spatial knowledge encoded in their internal representations. Find out more in the video teaser! Almost all VLMs prefer the egocentric relative FoR with reflected transform, similar to English. Yet, we reveal significant shortcomings of VLMs: notably, the models (1) exhibit poor robustness and consistency, (2) lack the flexibility to accommodate multiple FoRs, and (3) fail to adhere to language-specific or culture-specific conventions in cross-lingual tests, as English tends to dominate other languages. A shortened version will appear in Pluralistic Alignment Workshop Pluralistic Alignment Workshop #NeurIPS2024. It seems that the ArXiv moderators put it on hold and are eager to give it a thorough read first🤣! So here is the Paper/Code/Data: This collaboration turns out to be amazing, jointly led by Brian Zheyuan Zhang, @Hu_FY_ Jayjun Lee, with so many contributions and insights from Freda Shi, Parisa Kordjamshidi Michigan SLED Lab. With a growing effort to align vision-language models with human cognitive intuitions, we call for more attention to the ambiguous nature and cross-cultural diversity of spatial reasoning!

Martin Ziqiao Ma

35,595 просмотров • 1 год назад

Can you identify an oculogyric crisis? @ChristosGanos and colleagues teach us about the intriguing possibility that the rolling eyes we frequently see after administration of a drug, may be localized to a brain lesion network associated with a brain dopaminergic transcriptomic signature. Read about it in a new paper Brain. Key Points: - Oculogyric crises what are they? Acute episodes sustained, typically upward, conjugate deviation of the eyes. - Common cause? Acute D2-dopamine receptor blockade by a drug. - The authors used data from 14 previously reported cases of lesion-induced oculogyric crises and employed' lesion network mapping' to search for shared connections. - What was the common network? Basal ganglia, thalamic, brainstem nuclei and cerebellum. - A look at the gene expression profiles showed spatial overlap specifically with the gene coding for dopamine receptor type 2 (DRD2). - The very cool part: spatial overlap with DRD2 and DRD3 gene expression seemed to be 'specific to brain lesions associated with oculogyric crises when contrasted to lesions that led to other movement disorders.' My take: I love the use of multiple modalities to chase down a common neural network for oculogyric crises. We should keep in mind that these were cases associated with brain lesions, and as we all contribute by reporting more cases and sharing their MRI scans we will likely further refine our understanding of the network. For clinicians when you see this, look down that medication list for the 'dopamine blocker' as your first step in diagnosis. This video by Elan Louis provides a nice illustration of oculogyric crisis to help you to identify them in your practice. #Parkinsons #oculogyric
0:21

Can you identify an oculogyric crisis? @ChristosGanos and colleagues teach us about the intriguing possibility that the rolling eyes we frequently see after administration of a drug, may be localized to a brain lesion network associated with a brain dopaminergic transcriptomic signature. Read about it in a new paper Brain. Key Points: - Oculogyric crises what are they? Acute episodes sustained, typically upward, conjugate deviation of the eyes. - Common cause? Acute D2-dopamine receptor blockade by a drug. - The authors used data from 14 previously reported cases of lesion-induced oculogyric crises and employed' lesion network mapping' to search for shared connections. - What was the common network? Basal ganglia, thalamic, brainstem nuclei and cerebellum. - A look at the gene expression profiles showed spatial overlap specifically with the gene coding for dopamine receptor type 2 (DRD2). - The very cool part: spatial overlap with DRD2 and DRD3 gene expression seemed to be 'specific to brain lesions associated with oculogyric crises when contrasted to lesions that led to other movement disorders.' My take: I love the use of multiple modalities to chase down a common neural network for oculogyric crises. We should keep in mind that these were cases associated with brain lesions, and as we all contribute by reporting more cases and sharing their MRI scans we will likely further refine our understanding of the network. For clinicians when you see this, look down that medication list for the 'dopamine blocker' as your first step in diagnosis. This video by Elan Louis provides a nice illustration of oculogyric crisis to help you to identify them in your practice. #Parkinsons #oculogyric

Michael Okun

15,398 просмотров • 2 лет назад

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.
0:50

3D-LLM: Injecting the 3D World into Large Language Models paper page: Large language models (LLMs) and Vision-Language Models (VLMs) have been proven to excel at multiple tasks, such as commonsense reasoning. Powerful as these models can be, they are not grounded in the 3D physical world, which involves richer concepts such as spatial relationships, affordances, physics, layout, and so on. In this work, we propose to inject the 3D world into large language models and introduce a whole new family of 3D-LLMs. Specifically, 3D-LLMs can take 3D point clouds and their features as input and perform a diverse set of 3D-related tasks, including captioning, dense captioning, 3D question answering, task decomposition, 3D grounding, 3D-assisted dialog, navigation, and so on. Using three types of prompting mechanisms that we design, we are able to collect over 300k 3D-language data covering these tasks. To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism, 3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show that our model outperforms state-of-the-art baselines by a large margin (e.g., the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore, experiments on our held-in datasets for 3D captioning, task composition, and 3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative examples also show that our model could perform more tasks beyond the scope of existing LLMs and VLMs.

AK

249,798 просмотров • 3 лет назад

How does an embryo reliably "compute" its form - "cell by cell" - using only local interactions and mechanics, yet produce a precise global body plan? I’m excited to share our Nature Methods paper "MultiCell: geometric learning in multicellular development", presenting #AIxBiology research led by Haiqian Yang and the result of a great collaboration with Ming Guo, George Roy, Tomer Stern, Anh Nguyen and Dapeng Bi. A long-standing challenge in developmental biology is to predict how thousands of cells collectively self-organize as tissues fold, divide, and rearrange. In MultiCell, we represent a developing embryo as a dual graph that unifies two complementary views of tissue mechanics with single-cell resolution: cells as moving points (granular) and cells as a connected foam (junction network). This lets the model learn dynamics from both geometry and cell–cell connectivity. On whole-embryo 4D light-sheet movies of Drosophila gastrulation (~5,000 cells), our model predicts key cell behaviors and the timing of events, including junction loss, rearrangements, and divisions with high accuracy, at single-cell resolution. Beyond prediction, the same representation supports robust time alignment across embryos and offers interpretable activation maps that highlight the morphogenetic "drivers" of development. The broader goal is a foundation for cell-by-cell forecasting in more complex tissues, and eventually for detecting subtle dynamical signatures of disease. Kudos to the team for this inspiring collaboration with brilliant researchers to push the boundary of AI for biology! Citation: Yang, H., Roy, G., Nguyen, A.Q., Buehler, M.J., et al. MultiCell: geometric learning in multicellular development. Nature Methods (2025), DOI: 10.1038/s41592-025-02983-x Code/data links are in the manuscript.
0:46

How does an embryo reliably "compute" its form - "cell by cell" - using only local interactions and mechanics, yet produce a precise global body plan? I’m excited to share our Nature Methods paper "MultiCell: geometric learning in multicellular development", presenting #AIxBiology research led by Haiqian Yang and the result of a great collaboration with Ming Guo, George Roy, Tomer Stern, Anh Nguyen and Dapeng Bi. A long-standing challenge in developmental biology is to predict how thousands of cells collectively self-organize as tissues fold, divide, and rearrange. In MultiCell, we represent a developing embryo as a dual graph that unifies two complementary views of tissue mechanics with single-cell resolution: cells as moving points (granular) and cells as a connected foam (junction network). This lets the model learn dynamics from both geometry and cell–cell connectivity. On whole-embryo 4D light-sheet movies of Drosophila gastrulation (~5,000 cells), our model predicts key cell behaviors and the timing of events, including junction loss, rearrangements, and divisions with high accuracy, at single-cell resolution. Beyond prediction, the same representation supports robust time alignment across embryos and offers interpretable activation maps that highlight the morphogenetic "drivers" of development. The broader goal is a foundation for cell-by-cell forecasting in more complex tissues, and eventually for detecting subtle dynamical signatures of disease. Kudos to the team for this inspiring collaboration with brilliant researchers to push the boundary of AI for biology! Citation: Yang, H., Roy, G., Nguyen, A.Q., Buehler, M.J., et al. MultiCell: geometric learning in multicellular development. Nature Methods (2025), DOI: 10.1038/s41592-025-02983-x Code/data links are in the manuscript.

Markus J. Buehler

388,322 просмотров • 8 месяцев назад

New short course: Practical Multi AI Agents and Advanced Use Cases with crewAI. Learn to build and deploy advanced agent-based systems in real applications in this course, created with CrewAI and taught by its founder, João Moura! (Disclosure: I've made a small seed investment in CrewAI.) In this course, you’ll learn how to create advanced agent-based apps that use external tools, do performance testing, can be trained with human feedback, and perform multiple tasks with different large language models. You will build several practical agentic apps that provide real business value, such as an automated project planning system, lead scoring and engagement pipeline, customer support data analysis, and a robust content creation system. In detail, you will learn how to: - Create these multi-agent systems with the building blocks of tasks, agents, and crews, along with the different things that make them work, such as caching, memory, and guardrails. - Integrate your multi-agent application with internal and external systems. - Connect multiple agents in complex setups, including parallel, sequential, and hybrid configurations, and create flows involving multiple agentic applications working together. - Test your agentic workflow and train it using human feedback to optimize its performance for better and more consistent results. - Work with multiple LLMs in your multi-agent system, using the appropriate model sizes and providers to fit each agent’s specific task. - Start a project from scratch in your environment and prepare it for deployment. You’ll also learn from an interview between João and Jacob Wilson, the Commercial GenAI Principal at PwC , in which they discuss deploying agentic workflows in real industry use cases. By the end of this course, you will be equipped to start building custom multi-agentic systems for your work. Please sign up here!
2:52

New short course: Practical Multi AI Agents and Advanced Use Cases with crewAI. Learn to build and deploy advanced agent-based systems in real applications in this course, created with CrewAI and taught by its founder, João Moura! (Disclosure: I've made a small seed investment in CrewAI.) In this course, you’ll learn how to create advanced agent-based apps that use external tools, do performance testing, can be trained with human feedback, and perform multiple tasks with different large language models. You will build several practical agentic apps that provide real business value, such as an automated project planning system, lead scoring and engagement pipeline, customer support data analysis, and a robust content creation system. In detail, you will learn how to: - Create these multi-agent systems with the building blocks of tasks, agents, and crews, along with the different things that make them work, such as caching, memory, and guardrails. - Integrate your multi-agent application with internal and external systems. - Connect multiple agents in complex setups, including parallel, sequential, and hybrid configurations, and create flows involving multiple agentic applications working together. - Test your agentic workflow and train it using human feedback to optimize its performance for better and more consistent results. - Work with multiple LLMs in your multi-agent system, using the appropriate model sizes and providers to fit each agent’s specific task. - Start a project from scratch in your environment and prepare it for deployment. You’ll also learn from an interview between João and Jacob Wilson, the Commercial GenAI Principal at PwC , in which they discuss deploying agentic workflows in real industry use cases. By the end of this course, you will be equipped to start building custom multi-agentic systems for your work. Please sign up here!

Andrew Ng

341,204 просмотров • 1 год назад

New course to bring you up to state-of-the-art at using AI to help you code: Build Apps with Windsurf's AI Coding Agents, built in partnership with WIndsurf (Codeium) and taught by Anshul Ramachandran! AI-assisted IDEs (Integrated Development Environments) make developers’ workflows faster, more efficient, and much more fun. Agentic tools like Windsurf are more than just code autocomplete—they are collaborative coding agents that help you break down complex applications, iterate efficiently, and generate code that spans multiple files. Although a lot of coding assistants share the same underlying large language models for planning and reasoning, a major point of distinction is how they handle tools, keep track of context, and stay aligned with your intent as a developer. For instance, if you make modifications to a class definition in your code and make the same modifications to other classes in the same directory, you might tell the AI agent "Do the same thing in similar places in this directory." Here, tracking your intent means understanding that “the same thing" refers to that recent edit you just made, which must be followed by appropriate search and tool-calling to implement the changes. In this course, you'll learn the inner workings of coding agents, their strengths and limitations, and how to use Windsurf to quickly build several applications. In detail, you'll: - Build a mental model of how agents work by combining human-action tracking, tool integration, and context awareness to carry out an agentic coding workflow. - Learn the challenges of code search and discovery and how a multi-step retrieval approach helps coding agents address them. - Use Windsurf to analyze and understand a large, old codebase and update it to the latest versions of the frameworks and packages it uses. - Build a Wikipedia data analysis app that retrieves, parses, and analyzes word frequencies. - Enhance the performance of your Wikipedia analysis app by adding caching, and through this, also learn how to course-correct when the AI agent produces unexpected results. - Learn tips and tricks such as keyboard shortcuts, autocomplete, and @ mentions to quickly call on agentic capabilities. - Use image/multimodal capabilities of the AI agent to increase your development velocity; you'll see an example of uploading a mockup with sketched-out UI features, and ask the agent to use that to build new functionality to an app. By the end of this course, you’ll understand agentic coding in-depth and know how to use it to make your development process much faster, more efficient, and enjoyable. Please sign up here!
4:01

New course to bring you up to state-of-the-art at using AI to help you code: Build Apps with Windsurf's AI Coding Agents, built in partnership with WIndsurf (Codeium) and taught by Anshul Ramachandran! AI-assisted IDEs (Integrated Development Environments) make developers’ workflows faster, more efficient, and much more fun. Agentic tools like Windsurf are more than just code autocomplete—they are collaborative coding agents that help you break down complex applications, iterate efficiently, and generate code that spans multiple files. Although a lot of coding assistants share the same underlying large language models for planning and reasoning, a major point of distinction is how they handle tools, keep track of context, and stay aligned with your intent as a developer. For instance, if you make modifications to a class definition in your code and make the same modifications to other classes in the same directory, you might tell the AI agent "Do the same thing in similar places in this directory." Here, tracking your intent means understanding that “the same thing" refers to that recent edit you just made, which must be followed by appropriate search and tool-calling to implement the changes. In this course, you'll learn the inner workings of coding agents, their strengths and limitations, and how to use Windsurf to quickly build several applications. In detail, you'll: - Build a mental model of how agents work by combining human-action tracking, tool integration, and context awareness to carry out an agentic coding workflow. - Learn the challenges of code search and discovery and how a multi-step retrieval approach helps coding agents address them. - Use Windsurf to analyze and understand a large, old codebase and update it to the latest versions of the frameworks and packages it uses. - Build a Wikipedia data analysis app that retrieves, parses, and analyzes word frequencies. - Enhance the performance of your Wikipedia analysis app by adding caching, and through this, also learn how to course-correct when the AI agent produces unexpected results. - Learn tips and tricks such as keyboard shortcuts, autocomplete, and @ mentions to quickly call on agentic capabilities. - Use image/multimodal capabilities of the AI agent to increase your development velocity; you'll see an example of uploading a mockup with sketched-out UI features, and ask the agent to use that to build new functionality to an app. By the end of this course, you’ll understand agentic coding in-depth and know how to use it to make your development process much faster, more efficient, and enjoyable. Please sign up here!

Andrew Ng

139,978 просмотров • 1 год назад