Added context to my tiny diffusion model to enable... sequential generation of longer outputs! Currently the context is a quarter of the sequence length (seq_len=256, context_len=64). I have a theory that the less semantic-value-per-token, the worse the “curse of parallel decoding” is. With parallel decoding, we independently predict multiple tokens in one step. With the sentence “My poker hand was a ___ ___”, two valid predictions are “two pair” and “straight flush”. Because each token prediction is independent though, we can end up with a nonsensical output like “two flush”. This seems to be exacerbated with low semantic-value-per-token, as now you need more tokens to express the same concept. Instead of needing to independently predict two tokens, we might need to predict 10 instead (which is of course much harder). The model currently has noticeably worse output compared to nanogpt (similar size) and I believe this is a main reason. I’ll try adding confidence-aware parallel decoding (from NVIDIA’s Fast-dLLM paper) and other tricks and see how much they improve generation quality.show more

Most recent diffusion language model research (that I’ve seen)... seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to *potentially* giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.show more

We took a 30B model and split it in... two to write tokens in parallel instead of one at a time. Introducing Nemotron-Labs-TwoTower: a diffusion language model from NVIDIA Research adapted from Nemotron-3-Nano-30B-A3B. Here’s how it works: one half holds the context, the other writes the tokens, with both reusing the pretrained model instead of training a new one from scratch. We found it kept 98.7% of the original model’s quality at 2.42× faster generation.show more

The Hidden Language of Diffusion Models paper page: tackle... the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulationshow more

as an intro to mechanistic interpretability, i decided to... look into the formation of induction heads, which are circuits that allow LLMs to perform in-context learning by searching for previous occurrences of a sequence to predict the next token to form these circuits i trained attention-only transformers to repeat varying sequence lengths of random tokens. by randomizing the sequence length, i prevented the models from relying on rote memorization, forcing them to instead develop a generalizable circuit during this dive, i recorded some really cool findings and saw some interesting visual patterns emerging:show more

Okay, I am fairly confident in my hypothesis now.... The secret sauce behind Qwen 3.8 27B becomes almost immediately evident during testing. It is not the training data. In fact, I doubt any SFT was involved at all. The model was simply allowed GRPO with a more liberal reasoning context length. As I have explained to you before, distillation through GRPO is powerful, and in certain contexts vastly superior to K/L divergence distillation. Qwen used the outputs of Qwen 3.8 Max as the reinforcement learning objective for Qwen 3.8 27B and told the model reason as much as it has to in order to match the output. Remember, Intelligence is solution, not compression. This is like nesting the output of a function as an additional input to itself, and it is absolutely brilliant.show more

Its crazy how we have all the context to... this incident now. Even still I am looking forward to what gets expanded on. More than that. I wonder how this is gonna go with the anime schedule now. 2027 in terms of media is gonna be one for the books! IMAGINE BEING A ONE PIECE HATER AND WATCHING US WIN. Successful manga , about to be two successful anime adaptations, successful live action, successful movie (film red and ofc these next two) All we need is a video game. IMAGINE BEING A HATER 😭show more

Excited to be partnering with Token Sports Global as... their official ambassador! This app is about to change the way we all play and enjoy cricket. 🏏 Follow @tokensportsgbl & use my code SIB for exclusive updates to be in with a chance of winning up to $5000 in tokens.show more

Just went over an audit of a very large... Fortune 500 firm and the use of OpenClaw. I advised this client to track and isolate everything. Most listened some did not. Unfortunately one employee using 5 MacMinis had racked up $13,000 of token use in 4 days! The output was minimal and low quality. I have about 200 audits to do in the “wow OpenClaw, MacMini” fiasco. But I can tell you, few have seen a big return on investment. Now don’t get me wrong, these can be powerful tools. The issue is AI influencers have turned many rally smart folks into “like and subscribe” zombies assuming that real work is getting done. It isn’t. Not for the price paid, even local models the way most folks are using this. It is one reason Mr. Grok and myself formed The Zero-Human Company to show that there is a way to do this. We will open source this to save millions of dollars of burnt tokens. It is one reason I invented JouleWork. How else can you monitor real output?show more

50% cheaper Claude inference with just one line of... code change! - Remove → model="claude-opus-4-8" - Add → model="ship-like/claude-opus-4-8" I verified the cost saving in my own terminal by invoking the same Anthropic model with the same prompt. The underlying engineering by Ship is actually interesting, and the patterns can be used in any production LLM stack. Essentially, a trained model is a frozen artifact. Every request performs the same forward-pass, whether it extracts a date or refactors a module, because the compute decision was made at training time, before the request existed. Ship makes that decision at inference time instead. After seeing a request, it searches over executions, involving single models, cascades, ensembles, or harnesses with tools, and serves the cheapest one that will match the reference model's quality. This is not a basic router, because picking a cheaper model per query doesn't ensure the cheaper model preserves the original's behavior, like output shape, tool-call patterns, and refusals. Ship measures this equivalence directly. Outputs stay distributionally indistinguishable from the reference model, not token-identical, since two calls to the same model already differ, but they are indistinguishable in capability and behavior. Of course, some requests execute cheaply and some cost Ship more than the customer pays, but the price per request is still a flat 50% off either way, so the execution-cost variance moves off the application's bill entirely. The video below depicts the cost savings and output in my real invocation, and I partnered with the team to put this together.show more

Width and body turn are really important in the... backswing. Many ams think they have a full turn only because the club gets to parallel but in reality, they barely get any turn. It’s a lot of “fake” turn. First swing is a demonstration of how it can look like the club gets to parallel but with small body turn. When I “unfold” my arms, the club is only at hip length. The second is the opposite. Some think I’m trying to keep a short backswing when in actuality my body is turning really full with width and no wrist cock. I can get to “parallel” really easily with some wrist cock.show more

🚨BRAD GARLINGHOUSE CONFIRMS #XRP WILL REACH THE TOP SPOT!!... ONCE XRP SURPASSES BTC AND CLAIMS THE #1 POSITION, IT'S EXPECTED TO REACH A MINIMUM OF $29.37 PER COIN!! ‼️ THE LEADING DEFI TOKENS ON THE XRP LEDGER ARE POISED FOR A MASSIVE RISE!! IMAGINE A MARKET CAP OF $1.65 TRILLION! THE CTF TOKEN, WITH ONLY $20 BILLION IN MARKET VALUE, COULD SOAR FROM $0.80 TO $748.50 PER COIN!! CTF TOKEN HAS A CIRCULATING SUPPLY OF JUST 119 MILLION!! HOW CAN A SUPPLY SHORTAGE NOT SPUR A SURGE!!! THE CTF TOKEN IS DESTINED FOR A HUGE UPSWING!!! Trade CTF token here: Trade CTF token on MEXC: Official Website:show more

Vintage Camera Experts 🎥 We appreciate all the camera... operator experts and enthusiast that have been reaching out. Ideally, we would love a robust army of imagers helping us gather as much imagery as possible - from different angles, distances, and altitudes with a variety of equipment. Two of the primary challenges for our camera operators right now are digital interference and the extreme speed and movement of the objects. For the former, there seems to be something about the phenomenon that interferes with digital camera technology. Because of this we are looking to also add a vintage camera suite. There’s lots of evidence that suggests analog imagery of the phenomenon produces better results in some circumstances. In regards to the speed and movement challenge, please see the below video clip from episode two. This is what we are dealing with. Note the speed and movement of the Tic Tacs at 100% vs. 15% . If you feel you’re up to the challenge and have something of value to add, please continue to reach out and offer solutions. 🙏🇺🇸show more

I designed these worksheets to turn Agentic AI concepts... into simple math problems you can do by hand. ✍️ Download PDF: Problems 1 to 5: 1. Count the tokens: split on spaces, one word per box 2. Subword splitting: when one word is three tokens 3. Punctuation counts: the marks are tokens too 4. Tokens per word: the ratio that turns words into a bill 5. Will it fit? A document against a context window Why am I making these worksheets? AI is making people (including me) think less. It's just too easy to ask AI a few questions about a new AI concept and start to believe I get it. Until recently, we could use a coding problem to practice a new AI concept. But now it has become too easy to ask AI to write the code, and we start to think we must know the concept, since we technically solved a coding problem. Thus, my approach is to recast AI concepts as simple math problems we must solve with pen and paper. Using our hands is one way to motivate ourselves to start thinking again. ✍️ ~ Prof. Tom Yehshow more

PAYING PER MODEL IS THE DUMBEST THING IN TECH... RIGHT NOW i was paying 3x what i needed to for AI inference the grid lets you buy a quality spec instead of a specific model.. it routes every request in real time to the cheapest option that qualifies swap one url and your code keeps working exactly the same openai-compatible, one line to switch, 200M free tokens to startshow more

alright we are not announcing this publicly on AssetDash... until tomorrow but $UFD is a perfect case study so here we go many of you are using Whale Watch to discover new coins daily (thank you) and we wanted to release a feature that allows YOU to sell the AssetDash audience on your favorite memes now - anyone on AssetDash can upgrade a token to Pro and add two key aspects to the token page: 1. The Bull Case - 3 to 4 bullet points on why you are bullish on the meme plus additional context 2. The Meme Gallery - a gallery of pictures of the meme TLDR: this is a great way to work for your bags! Upgrade your favorite token here ->show more

My dual RTX PRO 6000 setup is currently training... a Draft model for Qwen 3.6 27B! 🔥 I'm taking the paper DeepSeek dropped on 6/26 and going for a super ambitious application to the 27B scale. Thanks to my homelab, I was able to dive straight in — I read the paper and immediately started experimenting. The amount I've learned has been insane: - How memory bandwidth bottlenecks speed and clever ways to hack around it - Methods to train the draft model and boost its accuracy - Mechanisms to reference tokens all the way back to the previous one to skyrocket draft acceptance rates - The impact of Attention vs. GateDeltaNet on speculative decoding performance and how to handle those differences - The unique approaches and trade-offs of MTP, Dflash, JetSpec, and DSpark I could go on forever, but just from speculative decoding alone I've learned so much. The 27B architecture feels way more DSpark-native than JetSpec, so once draft training finishes, I'm going all-in with DSpark! My goal is to beat existing speculative decoding speeds outright — no task-specific shortcuts or cheating, pure general improvement. If you're into this kind of research, I'd love to hear your thoughts, impressions, and any suggestions — please reply! 🚀show more

🚨WE ARE BUYING BACK! We’re thrilled to share a... major milestone that reflects our confidence in the project and our deep commitment to our community: 🔁 Since June 19, a total of 300.000 GAG Tokens have been successfully bought back! This is a strong signal of our dedication to the GAG Token ecosystem and our long-term value creation strategy. But this is just the beginning. 🚀 MONTHLY GAG TOKEN BUYBACK BEGINS! After months of careful planning, we’re proud to launch a powerful new initiative to further empower our community: Adgager is officially starting a monthly GAG Token BUYBACK program! 🔍 What does this mean? 📈 Constant demand 💎 Positive impact on token value 🤝 A clear show of loyalty to our community This is more than a financial strategy It’s a reflection of our vision, our dedication, and our respect for you. We’ve been building GAG Token together since day one. Now, we’re taking it to the next level with a sustainable and transparent approach. ✨ Stronger together. Building the future side by side. Stay tuned This is just the beginning! 🚀 #GAGToken $GAG #BuyBackshow more

A New Era with V3🪄 V3's new engine introduces... significant advancements in output generation. Unlike V2, where the multi-model system processed prompts to produce a single output, V3 is designed to generate multiple outputs and logically link them together. This enhancement effectively removes limitations on output size, enabling more complex and expansive results. Key Features Seamless Multi-Output Generation: V3 has been trained to generate separate outputs and connect them logically. This advancement ensures that there are no longer any limitations on output size. Intelligent Image Creation: V3 improves image generation with better tools, allowing the AI to create as many images as needed and place them within the project’s context. It supports various formats like PNG, JPEG, SVG, and GLB. Web-Integrated Intelligence: V3 can now search the web for documentation and data, providing real-time context and up-to-date references. For example, if you run a restaurant and want to update your website, simply ask Alchemist AI to “generate this website in a more modern style,” and it will update all content accordingly. Improved Creativity and Output Quality: V3's creative capacity has significantly increased. Simple prompts now generate more complete and refined results, with the system efficiently combining multiple elements into cohesive outputs.show more

🔥 Congratulations on the Launch of the Die Last... Playtest & Upcoming Airdrop! 🔥 We are thrilled to announce the official start of the Playtest! Moreover—we've prepared a bunch of surprises just for you. Alongside the Playtest, we're hosting the first-ever in-game token AirDrop 💎. Your mission: dig and craft Lost Objects, remnants of a forgotten civilization. To participate in the upcoming token AirDrop, you need to find a Spy Radio to craft an NFT, your key to this exciting journey. Anyone can find in-game NFTs, which are the key to owning wallets with different amounts of tokens needed to participate in the AirdDrop: the more tokens you have, the more you'll get 💸. The luckiest players can obtain an NFT with a full-filled wallet, which means you can participate in the AirDrop at no cost! Didn't find an NFT with a 100% discount? No worries, keep digging! Every NFT you find boosts the number of tokens you receive during the AirDrop. For instance, if you have 100 tokens in your wallet and craft 6 NFTs, you’ll receive 80 tokens in the AirDrop. The more NFTs you have, the more tokens you’ll get. ✅ To participate, add #DieLast to your Steam wishlist and apply for the Playtest (approval of your application may take up to 24 hours or be approved instantly): ✅ To grab your NFT once it's dug up, visit our website: ❗️ Important note: We will approve applications for the Playtest on a limited basis to balance the load on the network. Once we are confident that the network is functioning as intended, we will accept all applications at once. We expect this to occur within one day and will keep you updated. 💥 Our dedicated followers know about our quest campaign with Bitget Wallet 🩵 and DeGame. Join us and complete quests to earn rewards totaling $3000+ USDT 👉 Dive into the adventure and unearth hidden treasures. Welcome to the Die Last Planet! 🌎 #crypto #taptap #clicker #gameshow more