Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

AlphaFold by hand✍️ Excel ~ I designed this exercise to show (1) MSA multi-head attention, (2) Pair triangular update, two key components of the EvoFormer architecture.👇Join the AI Math community. Download xlsx.

104,965 görüntüleme • 1 yıl önce •via X (Twitter)

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Single vs Multi-hand Attention by hand ✍️ Resize matrices yourself 👉 The most important fact about multi-head attention: it has the same parameter count as single-head attention. The difference is purely structural — same total Wqkv weights, partitioned into smaller q–k–v triples. Look at the two diagrams below. Both Wqkv matrices have the same height — same number of weight rows, same number of parameters. What changes is how that single tall block is sliced. • Left. One head. The full Wqkv produces one big QKV: a tall Q (36 rows), a tall K, a tall V. One scoring computation runs over those full-width tensors. • Right. 3 heads. The same-height Wqkv is sliced into 3 smaller q–k–v triples — each 12 rows tall. 3 scoring computations run in parallel, each a thinner version of the left. The compute trade-off — kind of. Same Wqkv weights. Multi-head runs the attention scoring S = Kᵀ × Q once per head, so the dot-product count multiplies by H. • Single-head: seq × seq = 40² = 1600 dot products • Multi-head: seq × seq × H = 40² × 3 = 4800 dot products (3×) But each multi-head dot product is narrower — its inner dimension is head_dim instead of H × head_dim. So when you count actual scalar multiplications, the totals are equal: • Single-head: seq² × (H × head_dim) = 40² × 36 = 57600 • Multi-head: seq² × H × head_dim = 40² × 3 × 12 = 57600 Same FLOPs. Multi-head buys you H independent attention patterns at no extra weight cost and no extra arithmetic cost — it's the same total compute, sliced into H finer-grained heads.

Tom Yeh

35,116 görüntüleme • 1 ay önce

HOLY CRAP! I can't tell you how big this is for the medical community and drug discovery: Google Announces AlphaFold 3 AI. Details: Enhanced Molecular Prediction: AlphaFold 3 predicts the structure and interactions of all life's molecules, including proteins, DNA, RNA, ligands, and more, with unprecedented accuracy. Improved Interaction Accuracy: For protein interactions with other molecule types, AlphaFold 3 offers at least a 50% improvement over existing methods, and doubles the accuracy for some critical interactions. Transformative Potential for Science and Medicine: The model aims to deepen our understanding of biological processes and significantly advance drug discovery efforts. Accessibility for Researchers: AlphaFold 3's capabilities are largely accessible for free via the AlphaFold Server, providing an essential tool for scientific research. Drug Design Innovation: AlphaFold 3 is utilized by Isomorphic Labs in collaboration with pharmaceutical companies to accelerate drug design, potentially leading to new treatments for various diseases. Foundation in AlphaFold 2: Building on the breakthroughs of AlphaFold 2, this version extends its scope beyond proteins to a wide range of biomolecules, enhancing its utility in scientific research and application. Global Accessibility and Educational Support: The AlphaFold Server is a free platform for non-commercial research worldwide, supported by educational resources to foster wider adoption and innovation. Empowering Rapid Scientific Advancements: By making detailed molecular interactions easily accessible, AlphaFold 3 enables faster hypothesis testing and could reduce the time and cost typically associated with experimental protein-structure prediction. Responsible Development and Deployment: DeepMind has engaged with domain experts to assess the impacts and potential risks of AlphaFold, ensuring its responsible use in the scientific community. Broad Implications for Biology:AlphaFold 3 helps reveal complex cellular mechanisms and interactions, offering insights that could lead to improved agricultural crops, enhanced understanding of diseases, and novel therapeutic strategies.

Brian Krassenstein

258,536 görüntüleme • 2 yıl önce