Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Introducing SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Pre-Training Corpora What lies within a trillion-scale pre-training corpus? Can you truly guarantee your benchmarks are uncontaminated simply because there are no exact string matches? Alongside several research institutions in Japan, Sakana AI is proud to have collaborated...

104,363 görüntüleme • 8 ay önce •via X (Twitter)

9 Yorum

Sakana AI profil fotoğrafı
Sakana AI8 ay önce

SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora

Sami (e/acc) profil fotoğrafı
Sami (e/acc)8 ay önce

Here's a little summary for those who don't want to read the whole paper

cain1517 — e/acc ⏩ profil fotoğrafı
cain1517 — e/acc ⏩8 ay önce

wouldn't this be useful for context at inference (RAG)?

ノイズハッカー profil fotoğrafı
ノイズハッカー8 ay önce

1兆トークンの深淵に隠された「意味的な剽窃」をミリ秒単位で引きずり出す、冷酷なまでのデータ監査。 SoftMatcha 2は、LLMが「知能」と称して提示する回答の多くが、実は学習データの巧妙な「言い換え(変奏)」に過ぎないことを証明する、知の真贋鑑定機である。

norn profil fotoğrafı
norn8 ay önce

the uncomfortable implication is that most published benchmark numbers are measuring the test set's leakage profile, not the model's capability. this should be mandatory

Philippe Tremblay profil fotoğrafı
Philippe Tremblay8 ay önce

wow, this is fast!!

Lizzy profil fotoğrafı
Lizzy8 ay önce

@hardmaru SoftMatcha sounds like the most polite way to filter training data I've heard all week. Does it actually improve over hard masking, or is it just... softer on the model's feelings? 🤔

ikan laut profil fotoğrafı
ikan laut8 ay önce

this is very interesting

Gregor profil fotoğrafı
Gregor8 ay önce

You're right to question the robustness of benchmarks in trillion-scale pre-training corpora. What if the contaminants are not exact strings, but subtle patterns or biases that seep into the data through more insidious means, like data preprocessing or sampling strategies?

Benzer Videolar

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

The All-In Podcast

56,638 görüntüleme • 1 ay önce