Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Excited to release METAGENE-1, a 7B parameter metagenomic foundation model, built to aid in pathogen detection & pandemic monitoring. Pretrained on 1.5 trillion base pairs of DNA/RNA sequenced from wastewater. A collab w/ USC, Prime Intellect, & the Nucleic Acid Observatory. 🧵

13,390 görüntüleme • 1 yıl önce •via X (Twitter)

12 Yorum

Willie Neiswanger profil fotoğrafı
Willie Neiswanger1 yıl önce

Metagenomic sequencing of wastewater produces vast amounts of data that can capture public health trends at a societal scale. Our goal is to train a model on this data to help in large-scale wastewater monitoring & detection of novel bio threats. 🌐

Willie Neiswanger profil fotoğrafı
Willie Neiswanger1 yıl önce

Our data pipeline is: human microbiome > wastewater > metagenomic sequences > tokens > training data. Wastewater provides a rich source of data from tens of thousands of species across the human-adjacent microbiome. In total we pretrain on over 1.5T base pairs of DNA/RNA.

Willie Neiswanger profil fotoğrafı
Willie Neiswanger1 yıl önce

METAGENE-1 shows state-of-the-art results on pathogen detection, metagenomic embedding, and other genomic tasks. We also release new benchmarks for genomic detection and embedding (eg, Gene-MTEB, based on MTEB for LLMs). See our paper for details:

Willie Neiswanger profil fotoğrafı
Willie Neiswanger1 yıl önce

We leverage the ecosystem of modern LLM tooling—in tokenization, model architecture, training, infra, etc—for performance and extensibility. METAGENE-1 is standardized & easy to use. Hugging Face: Github:

Willie Neiswanger profil fotoğrafı
Willie Neiswanger1 yıl önce

Our paper also contains an in-depth discussion on safety when releasing metagenomic models. s/o to great coauthors @olliezliu @samsja19 @johannes_hage @UpupWang @_jason136, & jefftk Looking for collaborators to build on this with us — please reach out!

Λbstract profil fotoğrafı
Λbstract1 yıl önce

RT @InstaMAT_io: 👀 Are you looking for an alternative to Adobe Substance tools? Do not miss out on InstaMAT from killer node-based material…

Anthony Gitter profil fotoğrafı
Anthony Gitter1 yıl önce

@PrimeIntellect Exciting work! For next steps, are you thinking about how this would compare to classic bioinformatics workflows (e.g. for pathogen detection? It could also be relevant to compare to the @tatta_bio gLM model and DGEB benchmark.

Willie Neiswanger profil fotoğrafı
Willie Neiswanger1 yıl önce

@PrimeIntellect @tatta_bio Thanks for the links! Yes, these are great ideas for next steps.

Alon Albalak profil fotoğrafı
Alon Albalak1 yıl önce

@USC @PrimeIntellect Wow, awesome work @willieneis and team! Congrats on the release 🎉

Willie Neiswanger profil fotoğrafı
Willie Neiswanger1 yıl önce

@USC @PrimeIntellect Thanks Alon!

Ben Lengerich profil fotoğrafı
Ben Lengerich1 yıl önce

@USC @PrimeIntellect Congrats Willie! Looks very cool.

Willie Neiswanger profil fotoğrafı
Willie Neiswanger1 yıl önce

@USC @PrimeIntellect Thanks Ben!

Benzer Videolar

DARPA—who developed the Operation Warp Speed Death Jabs (see tweet 2)—now wants to inject people with "protein complexes" that, when hit with "different wavelengths of light" will "synthesize specific DNA or RNA sequences on demand." DARPA = literal Nazis. Nuremberg for DARPA! ----------------Partial transcription of clip--------------- "Synthetic DNA and RNA molecules could enable a host of next-generation technologies crucial for national security and global well-being. However, current methods for creating these molecules from scratch face significant limitations in terms of their scale, complexity, and use of hazardous chemicals. DARPA's Generative Optogenetics Program. "GO seeks to harness the power of light to direct the synthesis of DNA and RNA directly within living cells. If successful, this high-risk, high-reward research program has the potential to revolutionize medicine, agriculture, and manufacturing. "GO aims to unlock an era of programmable biology where we can communicate with and direct cell behavior with unprecedented speed, precision, and reach via optical signals. Performers on GO will design protein complexes that, when expressed in a living cell, use light-triggered signaling to deliver genetic information, instructing cells to incorporate specific building blocks based on different wavelengths of light to synthesize specific DNA or RNA sequences on demand. "The GO program focuses on two main research objectives. The first is achieving the core capability of synthesizing DNA or RNA directly in living cells using optical signals. The second focuses on developing error-mitigation methods that ensure high-fidelity nucleic acid synthesis."

Sense Receptor

50,875 görüntüleme • 7 ay önce

Karpathy told Dwarkesh that a 1 billion parameter model, trained on clean data, could hit the intelligence of today's 1.8 trillion parameter frontier. That is a 1,800x compression claim. The math behind it is more defensible than it sounds. When researchers at frontier labs look at random samples from their training corpus, they see stock ticker symbols, broken HTML, forum spam, autogenerated gibberish. Not Wikipedia. Not the Wall Street Journal. The actual pretraining dataset is mostly noise, and the model is burning parameters to vaguely remember all of it. One estimate pegs Llama 3's information compression at 0.07 bits per token. Well-structured English carries around 1.5 bits per token of real information. The trillion-parameter model is holding a roughly 5% resolution image of the internet it trained on. So when a lab ships a 1.8 trillion parameter model, the overwhelming majority of those weights are handling rough memorization. They are compression overhead for a noisy training set, taking up capacity that could be doing reasoning instead. Karpathy's proposal is to separate the two. Build a cognitive core: a small model that contains only the algorithms for reasoning and problem-solving, stripped of encyclopedic memorization. Pair it with external memory the model queries when it needs a fact. A 1 billion parameter reasoner plus retrieval beats a 1.8 trillion parameter model trying to do both. The data already supports this direction. GPT-4o runs at roughly 200 billion parameters and outperforms the original 1.8 trillion GPT-4. Inference costs for GPT-3.5 level performance fell 280x between 2022 and 2024, driven almost entirely by smaller, cleaner, better-architected models. The trend line is pointing where Karpathy says it should. The real implication for anyone tracking the AI trade: data quality is the actual constraint. The companies winning the next phase will be the ones who figured out what to train on, and what to throw away.

Aakash Gupta

507,952 görüntüleme • 2 ay önce

This is how DNA turns coded information into functional proteins - the building blocks of the nanomachines that keep the cells in your body alive. This complex process highlights the sophisticated interconnected systems of Life which must all exist together from the beginning, or Life doesn't happen. First, an RNA molecule is copied from a short segment of DNA. Without the specifically ordered DNA information, RNA cannot form, proteins cannot be built, cells stop working, and life ceases to exist. Life is information first. Once the RNA Molecule is created, it gets ejected from the Polymerase where it was built, and it travels through a complex molecular machine called a Nuclear Pore Complex (NPC), which is an information recognition device that controls the flow of information in and out of a cell's nucleus. The NPC is highly complex - composed of about 500-1,000 protein subunits, derived from a set of about 35 distinct proteins. Without this molecular machine, there is no regulation for what goes in and out of the cell's nucleus, which would lead to catastrophic death for the cell. It must exist for cells to exist. Once the RNA Molecule passes through the NPC, it travels to the Ribosome, a 2-part chemical factory which reads the information on RNA and uses it to construct functional proteins using a specifically sequenced chain of amino acids. Once complete, this protein will then be sent to the section of the cell it belongs to integrate into another molecular machine and do its job. The Ribosome is another highly complex molecular machine - consisting of between 56-80 proteins. Without this molecular machines, proteins cannot be built. Proteins are the building blocks of every cell in every organism on Earth. Without Ribosomes, Life doesn't exist. If you're paying attention, you'll start to realize that Life relies on a highly sophisticated interdependent network of complex machines, which all rely on each other for the function of the system. DNA requires the cell for stability, but the cell requires the proteins for its structure and function, but those proteins require DNA and RNA to be built - it's a circle of necessary interdependence. Systems like this cannot be built by evolutionary processes, which requires that each piece of the process is built by gradual incremental means over lots of time. Without all the pieces there, from the beginning, none of it works. There is only one known source of complex & interdependent informational systems like those we find in life: and that is from Intelligence. Molecular Biology is the best and most obvious evidence of the Intelligent Design in Life.

Divinely Designed

62,517 görüntüleme • 6 ay önce