Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Added context to my tiny diffusion model to enable sequential generation of longer outputs! Currently the context is a quarter of the sequence length (seq_len=256, context_len=64). I have a theory that the less semantic-value-per-token, the worse the “curse of parallel decoding” is. With parallel decoding, we independently predict multiple...

89,040 görüntüleme • 9 ay önce •via X (Twitter)

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Most recent diffusion language model research (that I’ve seen) seems to be using masking as the noising process. It looks like, however, most closed-source models (Google Gemini Diffusion and possibly Inception Labs’ Mercury) use a different noising process, where instead of masking tokens, they replace them with different tokens (either with a random token or a semantically similar token). I wondered how they were getting such high throughput with the latter noising process, since I believed that optimizing inference with KVCache approximation would be more difficult (for various reasons). I visualized this noising process with tiny-diffusion and compared it to normal unmasking, and was very surprised to see how fast the generation “settles” into a reasonable output, and then only slightly refines afterwards, requiring much fewer steps in total. Unmasking (where tokens are never remasked, the typical implementation) is inherently limited in generation speed by the fact that an increase in tokens decoded per step leads to more errors due to the mismatch between individual and marginal token probability distributions we sample from. The token replacement noising process seems to have a much different set of characteristics. Because we sample each token per step, every token makes “progress” towards the final output each iteration (in addition to *potentially* giving other tokens more information in future steps). Generally, masking has outperformed other noising processes, which is probably why most research focused on it (using smaller models). But the paper referred to in the retweet shows that random replacement as a noising process may scale better as model size increases. Big labs might have noticed these results much earlier (due to having drastically more training resources and being able to test larger models), which may explain the discrepancy in the choice of noising process. I’m gonna test this with larger models, since tiny-diffusion only has 10M parameters.

nathan (in sf)

40,440 görüntüleme • 6 ay önce

🔥 Congratulations on the Launch of the Die Last Playtest & Upcoming Airdrop! 🔥 We are thrilled to announce the official start of the Playtest! Moreover—we've prepared a bunch of surprises just for you. Alongside the Playtest, we're hosting the first-ever in-game token AirDrop 💎. Your mission: dig and craft Lost Objects, remnants of a forgotten civilization. To participate in the upcoming token AirDrop, you need to find a Spy Radio to craft an NFT, your key to this exciting journey. Anyone can find in-game NFTs, which are the key to owning wallets with different amounts of tokens needed to participate in the AirdDrop: the more tokens you have, the more you'll get 💸. The luckiest players can obtain an NFT with a full-filled wallet, which means you can participate in the AirDrop at no cost! Didn't find an NFT with a 100% discount? No worries, keep digging! Every NFT you find boosts the number of tokens you receive during the AirDrop. For instance, if you have 100 tokens in your wallet and craft 6 NFTs, you’ll receive 80 tokens in the AirDrop. The more NFTs you have, the more tokens you’ll get. ✅ To participate, add #DieLast to your Steam wishlist and apply for the Playtest (approval of your application may take up to 24 hours or be approved instantly): ✅ To grab your NFT once it's dug up, visit our website: ❗️ Important note: We will approve applications for the Playtest on a limited basis to balance the load on the network. Once we are confident that the network is functioning as intended, we will accept all applications at once. We expect this to occur within one day and will keep you updated. 💥 Our dedicated followers know about our quest campaign with Bitget Wallet 🩵 and DeGame. Join us and complete quests to earn rewards totaling $3000+ USDT 👉 Dive into the adventure and unearth hidden treasures. Welcome to the Die Last Planet! 🌎 #crypto #taptap #clicker #game

Die Last

13,396 görüntüleme • 2 yıl önce

So, my opinion on what the Antarctic (Antarctica) Anomaly is that it's a type of frequency technology. It must be way more powerful than HAARP, as many have claimed it to be, because we would see these anomalies at other HAARP sites, and we don't, not like this. With that said, and I'm very much trying to avoid letting what I want it to be not play a part here, I think it is a technology that is being used either off the coast of Antarctica itself or Bouvet Island. A third possibility is an area just to the northwest of the island that looks odd. It's possible it is a sonar scan from a ship, but why in that remote location? It looks like an antenna set up or rows of something that is out of place. I also believe that the weather events and fires that have taken place in Africa could possibly have been because of this. Each time we saw the anomaly, it was followed by a destructive weather event in Africa. A weird connection to that is we have been told and warned of a very busy 2024 Atlantic hurricane season. This is in part because of the above-average Atlantic ocean temperatures, which is the fuel to Hurricanes. With all this info, it's possible to see how the Anomaly could be a frequency tech that can manipulate or create weather, And or WARM up the Ocean temps to purposely enhance the Hurricane season and Storm growth. Keep in mind that many of our hurricanes and many of the biggest hurricanes have come from the west coast of Africa and form over the Cape Verde islands before heading towards the Caribbean and the United States. This is all of course speculation, and I'm learning many new things every day, so this idea may morph over time as we learn more. In the end, it is very hard to ignore all these findings. #antarctica #anonaly #AntarcticaAnomaly #BouvetIsland

In2ThinAir

442,580 görüntüleme • 2 yıl önce