Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens...

40,440 Aufrufe • vor 9 Monaten •via X (Twitter)

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

This weekend I invented something new, a new type of token using Uniswap v4 hooks that I call a Pair Token. A Pair Token is a new type of ERC-20. It is a fused, auto-compounding asset, a fusion of two tokens. For example, the stock tokens NVDA and SPCX fused into a single tokenized version of the two, with the ticker NVDASPCX, or ETH with NVDA, creating ETHNVDA. Each Pair Token represents a full-range Uniswap v4 liquidity pool. So why mint these tokens? These tokens are self-multiplying in terms of value, as at any moment anyone can call the compound function to compound the LP fees generated by the pair’s trading activity back into the asset. The token represents a new financial primitive, as it represents a proportional value and moves as an LP position NFT would, except that it is auto-compounding and, thanks to its v4 hook, allows for compounding and self-growth. Since it is an ERC-20, this token can be freely traded onchain, LPed, or even paired with a launchpad. A new type of token for a new type of exposure and strategy. Starting tonight, you’ll be able to mint or redeem Pair Tokens for any pair on Robinhood Chain via the StonkBrokers website. Minting and redeeming are free, and the compounding process on the tokens incurs a 10% fee on the LP fees generated, which is used for $STONKBROKER buybacks. I look forward to a new line of products and protocols on Robinhood Chain using this new financial primitive. Only on Robinhood Chain, powered by Uniswap. Hookr

SimpleFarmer

272,580 Aufrufe • vor 15 Tagen

Transformer and Mixture of Experts, explained visually! Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet? To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis. Read it below.

Daily Dose of Data Science

55,603 Aufrufe • vor 1 Monat

Massive update for AI Engineers! Training diffusion models just got a lot easier. dLLM is an open-source library that does for diffusion models what Hugging Face did for transformers. Here's why this matters: Traditional autoregressive models generate text left-to-right, one token at a time. Diffusion models work differently - they refine the entire sequence iteratively, giving you better control over generation quality and more flexible editing capabilities. The problem? Building and training these models required stitching together scattered tools and reimplementing research papers from scratch. Most of the tooling has been scattered and is hard to reproduce. dLLM changes this: It unifies everything you need to train, evaluate, and deploy diffusion language models: ↳ Scalable training with LoRA, DeepSpeed, and FSDP support ↳ Unified evaluation that abstracts away inference complexity ↳ Ready-to-use recipes for pretraining, finetuning, and evaluation The library includes implementations of models like LLaDA and Dream, plus training algorithms like Edit Flows that enable insertion, deletion, and substitution operations. The team just released ModernBERT-Chat models showing you can turn BERT into lightweight chatbots through masked instruction tuning. This is practical and worth exploring. The setup is straightforward. Run locally with Accelerate or scale to multi-node clusters with Slurm. If you're working with language models and want to explore diffusion-based approaches without rebuilding infrastructure, dLLM gives you a production-ready starting point. Link to the repo in the next tweet.

Akshay 🚀

64,011 Aufrufe • vor 11 Monaten

LongWriter Unleashing 10,000+ Word Generation from Long Context LLMs discuss: Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability.

AK

50,995 Aufrufe • vor 2 Jahren

Crypto narratives tend to move in cycles. 2020 was DeFi. 2021 became NFTs. 2023 turned into the AI boom. 2024–2025 were dominated by memecoins and attention tokens. But markets eventually rotate back to something simple: real revenue. That’s why some people are starting to look at iGaming tokens as a potential emerging narrative in 2026. Unlike many hype driven tokens, the iGaming sector already runs large cash flow businesses. Many platforms generate hundreds of millions of dollars in monthly revenue, yet their tokens often trade with far less volume than projects that barely produce revenue at all. In other words, there’s a visible mismatch between actual business activity and token market valuation. One ecosystem that sits right inside this discussion is 1win Token, which already operates as one of the top 10 online casinos globally by scale and user activity. The upcoming $1win Token is designed to connect that existing business with on chain incentives. Its token model includes buybacks and burns funded directly from casino revenue, tying token supply mechanics to real cash flow. There’s also an interesting structural difference compared to previous gaming tokens. For example, $RLB (Rollbit) saw a massive post launch rally, but the product and revenue scale at launch were significantly smaller than what 1win operates today. Another notable point is the launch design: instead of only farming an airdrop, 1win Token plans a public sale model, allowing broader participation from the start. If Web3 narratives are indeed shifting away from pure attention cycles and back toward revenue generating platforms, sectors like iGaming may start attracting more analytical focus and 1win could emerge as the biggest winner.

BitBull

20,972 Aufrufe • vor 7 Monaten