正在加载视频...

视频加载失败

Introducing SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Pre-Training Corpora What lies within a trillion-scale pre-training corpus? Can you truly guarantee your benchmarks are uncontaminated simply because there are no exact string matches? Alongside several research institutions in Japan, Sakana AI is proud to have collaborated...

104,363 次观看 • 8 个月前 •via X (Twitter)

9 条评论

Sakana AI 的头像
Sakana AI8 个月前

SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora

Sami (e/acc) 的头像
Sami (e/acc)8 个月前

Here's a little summary for those who don't want to read the whole paper

cain1517 — e/acc ⏩ 的头像
cain1517 — e/acc ⏩8 个月前

wouldn't this be useful for context at inference (RAG)?

ノイズハッカー 的头像
ノイズハッカー8 个月前

1兆トークンの深淵に隠された「意味的な剽窃」をミリ秒単位で引きずり出す、冷酷なまでのデータ監査。 SoftMatcha 2は、LLMが「知能」と称して提示する回答の多くが、実は学習データの巧妙な「言い換え(変奏)」に過ぎないことを証明する、知の真贋鑑定機である。

norn 的头像
norn8 个月前

the uncomfortable implication is that most published benchmark numbers are measuring the test set's leakage profile, not the model's capability. this should be mandatory

Philippe Tremblay 的头像
Philippe Tremblay8 个月前

wow, this is fast!!

Lizzy 的头像
Lizzy8 个月前

@hardmaru SoftMatcha sounds like the most polite way to filter training data I've heard all week. Does it actually improve over hard masking, or is it just... softer on the model's feelings? 🤔

ikan laut 的头像
ikan laut8 个月前

this is very interesting

Gregor 的头像
Gregor8 个月前

You're right to question the robustness of benchmarks in trillion-scale pre-training corpora. What if the contaminants are not exact strings, but subtle patterns or biases that seep into the data through more insidious means, like data preprocessing or sampling strategies?

相关视频

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

The All-In Podcast

56,638 次观看 • 1 个月前