Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Introducing SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Pre-Training Corpora What lies within a trillion-scale pre-training corpus? Can you truly guarantee your benchmarks are uncontaminated simply because there are no exact string matches? Alongside several research institutions in Japan, Sakana AI is proud to have collaborated... in the development of SoftMatcha 2, an ultra-fast and flexible search tool that enables search over trillion-scale natural language corpora in under 0.3 seconds, even while handling semantic variations (substitution, insertion, and deletion). No existing tool meets all these criteria, including infini-gram-mini (EMNLP’25 Best Paper) or the original SoftMatcha (ICLR’25). Our approach employs string matching based on suffix arrays that scales well with corpus size. To mitigate the combinatorial explosion induced by the semantic relaxation of queries, our method is built on two key algorithmic ideas: fast exact lookup enabled by a disk-aware design, and dynamic corpus-aware pruning. As a practical application, we demonstrate that SoftMatcha 2 identifies potential benchmark contamination in pre-training corpora that existing exact-match approaches miss. You can try searching through a 100B-scale corpus via our online demo. The system remains blazingly fast even on trillion-token corpora, so we encourage you to host it yourself for larger scales. Demo: Paper: Code: This work is a collaboration with researchers from the University of Tokyo, NII, Kyoto University, SOKENDAI, NINJAL, Tohoku University, and RIKEN.show more

Sakana AI

139,895 subscribers

104,363 views • 8 months ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

9 Comments

Sakana AI8 months ago

SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora

Sami (e/acc)8 months ago

Here's a little summary for those who don't want to read the whole paper

cain1517 — e/acc ⏩8 months ago

wouldn't this be useful for context at inference (RAG)?

ノイズハッカー8 months ago

1兆トークンの深淵に隠された「意味的な剽窃」をミリ秒単位で引きずり出す、冷酷なまでのデータ監査。 SoftMatcha 2は、LLMが「知能」と称して提示する回答の多くが、実は学習データの巧妙な「言い換え（変奏）」に過ぎないことを証明する、知の真贋鑑定機である。

norn8 months ago

the uncomfortable implication is that most published benchmark numbers are measuring the test set's leakage profile, not the model's capability. this should be mandatory

Philippe Tremblay8 months ago

wow, this is fast!!

Lizzy8 months ago

@hardmaru SoftMatcha sounds like the most polite way to filter training data I've heard all week. Does it actually improve over hard masking, or is it just... softer on the model's feelings? 🤔

ikan laut8 months ago

this is very interesting

Gregor8 months ago

You're right to question the robustness of benchmarks in trillion-scale pre-training corpora. What if the contaminants are not exact strings, but subtle patterns or biases that seep into the data through more insidious means, like data preprocessing or sampling strategies?

Related Videos

The infrastructure for working with video data at scale barely exists. Grass Video Search is our first step in changing that. You can now find anything in a video based on the contents of its frames, not just transcripts or tags. Over the last several months, the Grass Foundation has quietly assembled one of the largest video datasets in history: over a billion videos from across the public web, and is segmenting them by scene with VLM annotations of keyframes. This dataset is now searchable, so you can find the exact moment a visual, pattern, or concept appears. Curation has always been the bottleneck in video analysis. We built a system that automates it. This is just the beginning. Over 4 billion additional videos are queued for processing, and the index is growing daily. We’re opening access to a limited set of collaborators. If you’re a researcher, journalist, or nonprofit working with video, archives, or large-scale analysis, we’d love to hear from you.

The infrastructure for working with video data at scale barely exists. Grass Video Search is our first step in changing that. You can now find anything in a video based on the contents of its frames, not just transcripts or tags. Over the last several months, the Grass Foundation has quietly assembled one of the largest video datasets in history: over a billion videos from across the public web, and is segmenting them by scene with VLM annotations of keyframes. This dataset is now searchable, so you can find the exact moment a visual, pattern, or concept appears. Curation has always been the bottleneck in video analysis. We built a system that automates it. This is just the beginning. Over 4 billion additional videos are queued for processing, and the index is growing daily. We’re opening access to a limited set of collaborators. If you’re a researcher, journalist, or nonprofit working with video, archives, or large-scale analysis, we’d love to hear from you.

Grass

471,910 views • 1 year ago

I cant believe this guy just made a permanent solution to context bloat and open sourced it all! when we tested this tool (Context+) for solving an issue on the OpenCode repository, the agent using this tool used ~6.5k fewer tokens, found the code and fixed it in half the time! the results were surprising: 6 to 10k tokens saved per prompt, completed task in ~2 minutes while the agent running without the tool took ~4 mins for the same and got stuck in loops bro built an entire beast by using all the modern tools that we could think of: undo trees, semantic search by meaning (by haskellforall), advanced refactoring, blast radius, advanced file context trees, restore points... i can keep going on semantic code search and context trees are the future of agentic coding and this tool proves it the feature i loved the most is semantic search and how it gets things done 2x faster with least possible tokens it makes an agent that actually knows what it’s doing and not just guessing, it makes meaning from your code similar to RAG. if you aren't optimizing your context, you are just burning money the developer says this tool is still under development, it can have unexpected behavior and the docs need updates but the video shows the reality of how fast it can be github: get here:

I cant believe this guy just made a permanent solution to context bloat and open sourced it all! when we tested this tool (Context+) for solving an issue on the OpenCode repository, the agent using this tool used ~6.5k fewer tokens, found the code and fixed it in half the time! the results were surprising: 6 to 10k tokens saved per prompt, completed task in ~2 minutes while the agent running without the tool took ~4 mins for the same and got stuck in loops bro built an entire beast by using all the modern tools that we could think of: undo trees, semantic search by meaning (by haskellforall), advanced refactoring, blast radius, advanced file context trees, restore points... i can keep going on semantic code search and context trees are the future of agentic coding and this tool proves it the feature i loved the most is semantic search and how it gets things done 2x faster with least possible tokens it makes an agent that actually knows what it’s doing and not just guessing, it makes meaning from your code similar to RAG. if you aren't optimizing your context, you are just burning money the developer says this tool is still under development, it can have unexpected behavior and the docs need updates but the video shows the reality of how fast it can be github: get here:

forloop

226,544 views • 7 months ago

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

David Friedberg: Frontier Models are Training on Your Novel Insights as “De-Identified Data” @jason: “Should they trust any of these LLMs with their proprietary knowledge for fear of having it cribbed into a core LLM?” david friedberg: “I have had experiences where we've asked some fairly novel scientific questions, and (the AI model) identifies it as a novel insight. It's like, ‘Oh, never thought about that, interesting, blah, blah, blah.’ And then using a different account, asking the next version (of the model) later, I've now experienced this. It's like, ‘Oh, well, you could do this,’ and it actually just describes this exact thing that we had in our chat in the previous version. Now, these are a handful of anecdotal experiences, but I know the domain that we work in, and the niche of it, and the ideation of this stuff, and the novelty of this stuff, and the lack of papers being published, and so on. So I know that there isn't some new corpus of information out there that's training the new model. So all I can say at that point is that my conversation or our analyses have been used for training.” David Sacks: “Okay, this does raise a really good question. What does it mean that the model is allowed to train on unidentifiable data?” Friedberg: “Well, that's my point. So it doesn't use any of my personal information, but it can use an insight derived from our chat, which it can then say is some training data that is unrelated. But the truth is, it's actually a piece of IP that's our organization’s IP, and our engagement back and forth. We don't have any NDA or confidentiality provisions or protections with them being a service provider back to us. This is why I care a lot about open source because I don't want them having my chat logs because they can use it for training to create an IP advantage that is now diffused to the rest of the market.”

The All-In Podcast

56,638 views • 1 month ago

Today morning my supplementary question no 1 on Young India Skills University. I'm proud to say that under the able and dynamic leadership of our Hon'ble Chief Minister Revanth Reddy, he is ambitiously setting up the Young India Skills University in Hyderabad, with the objective of providing job opportunities to the students and imparting skill development training according to the requirements of the local industries that are in demand in collaboration with various companies in the PPP model. A total of 17 priority sectors have been identified, and we will be providing training to 2 thousand people in the first year of 2025-26. We will train 20,000 people every year, and the state has allocated Rs. 100 crore on its part for the same and further wants to create a corpus fund of about Rs. 300-500 crores. Now my question to the ministry is if there is any proposal from the central government to adequately fund this Skills University. This will make it a grand success and will lead to the realisation of the goal of a developed nation by 2047. #YoungIndiaSkillUniversity #Telangana #TelanganaRising

Today morning my supplementary question no 1 on Young India Skills University. I'm proud to say that under the able and dynamic leadership of our Hon'ble Chief Minister Revanth Reddy, he is ambitiously setting up the Young India Skills University in Hyderabad, with the objective of providing job opportunities to the students and imparting skill development training according to the requirements of the local industries that are in demand in collaboration with various companies in the PPP model. A total of 17 priority sectors have been identified, and we will be providing training to 2 thousand people in the first year of 2025-26. We will train 20,000 people every year, and the state has allocated Rs. 100 crore on its part for the same and further wants to create a corpus fund of about Rs. 300-500 crores. Now my question to the ministry is if there is any proposal from the central government to adequately fund this Skills University. This will make it a grand success and will lead to the realisation of the goal of a developed nation by 2047. #YoungIndiaSkillUniversity #Telangana #TelanganaRising

Kiran Kumar Chamala

71,229 views • 1 year ago

VIDEO | Delhi: Professor Dr Aishwarya of Galgotias University responds over reports of the university being asked to vacate its stall at the AI Summit Expo, says, "No, we have not received any directive yet to vacate the expo area. In fact, we are ready to answer for what happened yesterday. A viral tweet circulated which went in a completely wrong direction and was misinterpreted. Everywhere, we have maintained the same claim—we never said that we manufacture or design these things. We have always said that our university is investing Rs 350 crore in AI. As part of that investment, we have brought different kinds of things into the university campus so that our students can explore, expand, and carry out research and development, and try to build an advanced prototype... actually it was misinterpreted. It is a very well-known fact, and we also know that this is something available everywhere on the internet. Why would we claim such a thing? It was misinterpreted, a viral tweet was posted and exaggerated—you know how such things happen. Therefore, we want to clarify that we are not a manufacturing unit. We are not people who make all these things. We are only here so that our university students can conduct research and development. As part of this investment, we are bringing these things in for that purpose.” #GalgotiasUniversity #IndiaAISummit2026 (Full video available on PTI Videos -

VIDEO | Delhi: Professor Dr Aishwarya of Galgotias University responds over reports of the university being asked to vacate its stall at the AI Summit Expo, says, "No, we have not received any directive yet to vacate the expo area. In fact, we are ready to answer for what happened yesterday. A viral tweet circulated which went in a completely wrong direction and was misinterpreted. Everywhere, we have maintained the same claim—we never said that we manufacture or design these things. We have always said that our university is investing Rs 350 crore in AI. As part of that investment, we have brought different kinds of things into the university campus so that our students can explore, expand, and carry out research and development, and try to build an advanced prototype... actually it was misinterpreted. It is a very well-known fact, and we also know that this is something available everywhere on the internet. Why would we claim such a thing? It was misinterpreted, a viral tweet was posted and exaggerated—you know how such things happen. Therefore, we want to clarify that we are not a manufacturing unit. We are not people who make all these things. We are only here so that our university students can conduct research and development. As part of this investment, we are bringing these things in for that purpose.” #GalgotiasUniversity #IndiaAISummit2026 (Full video available on PTI Videos -

Press Trust of India

83,453 views • 8 months ago

"I like to move extremely fast, but in order to move fast, you need to have brakes." Up to 200,000 AI agents wake up at Kavak every day. CPO & AI Officer Alejandro Maza says eval quality is what sets the ceiling on speed at that scale: "So to give you a sense of the scale, 96% of all interactions are handled by agents. No humans there. 95% of all transactions are completely handled by agents." "AI is super powerful, and I've seen many companies get this wrong because they try to go slow because they don't have the right brakes. So I thought about it the other way around: how fast can we go? Well, it depends on the quality of our evals." "A good rule of thumb here is we spend about the same amount of engineer time, tokens, and money on building the evals as building the agents." Alejandro Maza

"I like to move extremely fast, but in order to move fast, you need to have brakes." Up to 200,000 AI agents wake up at Kavak every day. CPO & AI Officer Alejandro Maza says eval quality is what sets the ceiling on speed at that scale: "So to give you a sense of the scale, 96% of all interactions are handled by agents. No humans there. 95% of all transactions are completely handled by agents." "AI is super powerful, and I've seen many companies get this wrong because they try to go slow because they don't have the right brakes. So I thought about it the other way around: how fast can we go? Well, it depends on the quality of our evals." "A good rule of thumb here is we spend about the same amount of engineer time, tokens, and money on building the evals as building the agents." Alejandro Maza

a16z

41,467 views • 2 months ago

Today we announced our new Fairwater datacenter in Atlanta, connected with our first Fairwater site in Wisconsin and our broader Azure footprint to create the world’s first AI superfactory. Fairwater exemplifies our vision for a fungible fleet: infra that can serve any workload, anywhere, on fit-for-purpose accelerators and network paths, with maximum performance and efficiency. AI workloads have evolved beyond large-scale pre-training. Today, they encompass fine-tuning, reinforcement learning (RL), synthetic data generation, evaluation pipelines, and more. Fairwater is built to support this full lifecycle: Max density: Fairwater’s two-story design and liquid cooling system lets us place racks in three dimensions and pack them with GPUs as densely as possible, minimizing cable runs and improving latency and effective bandwidth. Fleet: Each Fairwater DC can integrate hundreds of thousands of the latest NVIDIA GPUs into a single coherent cluster. This provides flexible infra that can support the full spectrum of workloads, and ensure no GPU is left unnecessarily idle. And that’s on top of the more than 100,000 GB300s coming online this quarter alone for inference across the rest of our fleet. For us, it’s all about turning every gigawatt into the maximum number of useful tokens. Not every GW is created equal! Planet-scale: Every Fairwater DC will connect through our continent-spanning AI WAN to prior generations of AI supercomputers, forming a truly fungible pool of compute. This enables developers to scale beyond the capacity of a single site and dynamically land workloads on the right infra for their needs. Together, these innovations let us bring together different generations of silicon and AI systems across DCs and geos into a single elastic system that scales seamlessly across training and inference workloads And this elastic AI capacity is all available alongside all the other cloud services (compute, storage, databases, app services) that AI agents and workloads need. This is what we mean when we talk about building a fungible fleet – a single, unified platform that pushes the limits of performance per watt and per dollar. Read more:

Today we announced our new Fairwater datacenter in Atlanta, connected with our first Fairwater site in Wisconsin and our broader Azure footprint to create the world’s first AI superfactory. Fairwater exemplifies our vision for a fungible fleet: infra that can serve any workload, anywhere, on fit-for-purpose accelerators and network paths, with maximum performance and efficiency. AI workloads have evolved beyond large-scale pre-training. Today, they encompass fine-tuning, reinforcement learning (RL), synthetic data generation, evaluation pipelines, and more. Fairwater is built to support this full lifecycle: Max density: Fairwater’s two-story design and liquid cooling system lets us place racks in three dimensions and pack them with GPUs as densely as possible, minimizing cable runs and improving latency and effective bandwidth. Fleet: Each Fairwater DC can integrate hundreds of thousands of the latest NVIDIA GPUs into a single coherent cluster. This provides flexible infra that can support the full spectrum of workloads, and ensure no GPU is left unnecessarily idle. And that’s on top of the more than 100,000 GB300s coming online this quarter alone for inference across the rest of our fleet. For us, it’s all about turning every gigawatt into the maximum number of useful tokens. Not every GW is created equal! Planet-scale: Every Fairwater DC will connect through our continent-spanning AI WAN to prior generations of AI supercomputers, forming a truly fungible pool of compute. This enables developers to scale beyond the capacity of a single site and dynamically land workloads on the right infra for their needs. Together, these innovations let us bring together different generations of silicon and AI systems across DCs and geos into a single elastic system that scales seamlessly across training and inference workloads And this elastic AI capacity is all available alongside all the other cloud services (compute, storage, databases, app services) that AI agents and workloads need. This is what we mean when we talk about building a fungible fleet – a single, unified platform that pushes the limits of performance per watt and per dollar. Read more:

Satya Nadella

908,861 views • 11 months ago

$AI agents × crypto rails is one of the most bullish narratives we have "It's happening so incredibly fast that I'm sure within six months, we're going to be giving these things literally everything. They're going to be running our lives, and we're going to wonder what to do with all of that new free time." "This all comes at the same time that OpenAI is raising at a $1.7 trillion valuation, and Anthropic at a $2 trillion valuation pre-IPO and in the terms and conditions saying the AIs are all coming to kill us. Maybe we shouldn't give the AI agents all of our information. I'm not really sure." "But the real question is, if your agent's trading for you on Robinhood, and that can be controlled through ChatGPT to create your strategies and manage your entire life, what happens when they actually need to spend money? Do you think they're going to KYC themselves on your bank account? No. They're going to use stablecoins and other tokens, and they're going to do all of this on blockchain rails." "We are about one tiny fraction of an inch away from these agents doing effectively everything in your life, which includes all of your transactions, and will include them transacting with one another on micropayments that have nothing to do with you. That's all going to happen on crypto, and it's one of the most bullish narratives we have."$

AI agents × crypto rails is one of the most bullish narratives we have "It's happening so incredibly fast that I'm sure within six months, we're going to be giving these things literally everything. They're going to be running our lives, and we're going to wonder what to do with all of that new free time." "This all comes at the same time that OpenAI is raising at a $1.7 trillion valuation, and Anthropic at a $2 trillion valuation pre-IPO and in the terms and conditions saying the AIs are all coming to kill us. Maybe we shouldn't give the AI agents all of our information. I'm not really sure." "But the real question is, if your agent's trading for you on Robinhood, and that can be controlled through ChatGPT to create your strategies and manage your entire life, what happens when they actually need to spend money? Do you think they're going to KYC themselves on your bank account? No. They're going to use stablecoins and other tokens, and they're going to do all of this on blockchain rails." "We are about one tiny fraction of an inch away from these agents doing effectively everything in your life, which includes all of your transactions, and will include them transacting with one another on micropayments that have nothing to do with you. That's all going to happen on crypto, and it's one of the most bullish narratives we have."

The Wolf Of All Streets

19,877 views • 18 days ago

David Friedberg reveals frontier labs are taking novel insights from his chat history and packaging them as their own "I have had experiences where we've asked some fairly novel scientific questions, and it identifies it as a novel insight, 'oh, never thought about that, might be interesting,' blah blah blah..." "And the using a different account, asking the next model version, it's like, 'oh, you could do this'. And it actually just describes this exact thing that we had in our chat in the previous version." "These are a handful of anecdotal experiences. But I know the domain that we work in, and the novelty of this stuff. So I know that there isn't some new corpus of information out there that's training the new model." "So all I can say at that point is that my conversation or our analyses have been used for training. The truth is, it's actually a piece of IP, that's our organizational IP. We don't have any NDA or confidentiality protections, with them being a service provider back to us." "This is why I care a lot about open source, because I don't want them having my chat logs, because they can use them for training to create an IP that is now diffused to the rest of the market. I find this very unfair."

David Friedberg reveals frontier labs are taking novel insights from his chat history and packaging them as their own "I have had experiences where we've asked some fairly novel scientific questions, and it identifies it as a novel insight, 'oh, never thought about that, might be interesting,' blah blah blah..." "And the using a different account, asking the next model version, it's like, 'oh, you could do this'. And it actually just describes this exact thing that we had in our chat in the previous version." "These are a handful of anecdotal experiences. But I know the domain that we work in, and the novelty of this stuff. So I know that there isn't some new corpus of information out there that's training the new model." "So all I can say at that point is that my conversation or our analyses have been used for training. The truth is, it's actually a piece of IP, that's our organizational IP. We don't have any NDA or confidentiality protections, with them being a service provider back to us." "This is why I care a lot about open source, because I don't want them having my chat logs, because they can use them for training to create an IP that is now diffused to the rest of the market. I find this very unfair."

dnap

380,411 views • 1 month ago

Searching 32 million records in 30 milliseconds is insane! typesense | is a crazy fast open-source search engine that supports natural language search. It's like Elasticsearch but much easier to set up, configure, and use. You can do all of this with Typesense: • Fuzzy search: Typo-tolerant fuzzy search-as-you-type • Vector search: Return records that are similar to each other • Semantic search: Return records that are conceptually related • Geo Search: Return records that are close to a given lat/long Typesense's Natural Language Search is pretty cool: It turns a sentence into a structured search query, then runs it against your data. For example: The query "Find me a BMW under $40,000, newer than 2020, with the lowest mileage first" will produce the following: • Filters: brand = BMW, price 2020 • Sort: mileage ascending Typesense will then apply these filters and sorting rules to the data. They offer client libraries in Python, JavaScript, PHP, Ruby, Go, Rust, Swift, .NET, Java, and pretty much any other programming language. You can start searching in less than a minute: 1. Run their Docker image (or one of their native binaries) 2. Import your data 3. Start searching! GitHub repository: I recorded a quick video to show you a demo. I promise you haven't seen anything this fast! Thanks to the team for partnering with me on this post.

Santiago

59,608 views • 10 days ago

The most valuable skill you can teach yourself right now is agent management. Nobody hands you a system for that, so here is mine. It runs at 22 to 25 pull requests a day, and sometimes 40. The agents scale without limit. They move fast, then they stop, because they need a yes, a no, or an actually-do-it-this-way from me. 1) Pin the threads that matter today. Everything else runs on its own and I get back to it when I can. 2) Check the pinned ones every 25 minutes. Slow is smooth and smooth is fast, and if I click through threads all day I get wiped out. 3) Keep the day's real priorities on paper. About 10 of those PRs are putting out fires, and the list is what stops the fires from eating the day. 4) Run one premium model as the parent thread and let it spin up children on a cheaper model. The catch: a few years ago you made 2 or 3 high-stakes decisions in a day. Now you make 10 to 20 before lunch, and you work a lot more. On the 18th I climbed Mount Washington with my son and had no signal, and I still shipped 8 PRs that morning while he was asleep. Your homework: do more than 50% of your work from your phone.

The most valuable skill you can teach yourself right now is agent management. Nobody hands you a system for that, so here is mine. It runs at 22 to 25 pull requests a day, and sometimes 40. The agents scale without limit. They move fast, then they stop, because they need a yes, a no, or an actually-do-it-this-way from me. 1) Pin the threads that matter today. Everything else runs on its own and I get back to it when I can. 2) Check the pinned ones every 25 minutes. Slow is smooth and smooth is fast, and if I click through threads all day I get wiped out. 3) Keep the day's real priorities on paper. About 10 of those PRs are putting out fires, and the list is what stops the fires from eating the day. 4) Run one premium model as the parent thread and let it spin up children on a cheaper model. The catch: a few years ago you made 2 or 3 high-stakes decisions in a day. Now you make 10 to 20 before lunch, and you work a lot more. On the 18th I climbed Mount Washington with my son and had no signal, and I still shipped 8 PRs that morning while he was asleep. Your homework: do more than 50% of your work from your phone.

The Startup Ideas Podcast (SIP) 🧃

11,736 views • 1 month ago

Today we’re releasing V-JEPA, a method for teaching machines to understand and model the physical world by watching videos. This work is another important step towards Yann LeCun’s outlined vision of AI models that use a learned understanding of the world to plan, reason and accomplish complex tasks. Details ➡️ We're releasing a collection of V-JEPA vision models trained with a feature prediction objective using self-supervised learning. The models are able to understand and predict what is going on in a video, even with limited information. It learns by predicting missing or obscured parts of a video in its internal feature space. Unlike generative approaches that fill in missing pixels, this flexible approach enables up to 6x improvements in training and sample efficiency. The models were pre-trained on entirely unlabeled data, and a small amount of labeled data can be used to train a task-specific prediction head on top after pre-training. Our results show that, using a frozen backbone, our top V-JEPA models achieve 82.0% on Kinetics-400, 72.2% on Something-Something-v2 and 77.9% on ImageNet1K — competitive with or exceeding previous leading video models. We believe that this work is an important milestone on the path to advancing machine intelligence.

Today we’re releasing V-JEPA, a method for teaching machines to understand and model the physical world by watching videos. This work is another important step towards Yann LeCun’s outlined vision of AI models that use a learned understanding of the world to plan, reason and accomplish complex tasks. Details ➡️ We're releasing a collection of V-JEPA vision models trained with a feature prediction objective using self-supervised learning. The models are able to understand and predict what is going on in a video, even with limited information. It learns by predicting missing or obscured parts of a video in its internal feature space. Unlike generative approaches that fill in missing pixels, this flexible approach enables up to 6x improvements in training and sample efficiency. The models were pre-trained on entirely unlabeled data, and a small amount of labeled data can be used to train a task-specific prediction head on top after pre-training. Our results show that, using a frozen backbone, our top V-JEPA models achieve 82.0% on Kinetics-400, 72.2% on Something-Something-v2 and 77.9% on ImageNet1K — competitive with or exceeding previous leading video models. We believe that this work is an important milestone on the path to advancing machine intelligence.

AI at Meta

704,200 views • 2 years ago

Can AI agents adapt zero-shot, to complex multi-step language instructions in open-ended environments? We present MaestroMotif, a method for AI-assisted skill design that produces highly capable and steerable hierarchical agents. To the best of our knowledge, it is the first method that, without expert labeled datasets, solves compositional tasks requiring hundreds of steps for completion. All the modules within MaestroMotif are learned from interaction: from the highest level of planning to the lowest-level of sensorimotor control. On the open-ended domain of NetHack, it surpasses existing approaches, including those that are fine-tuned specifically for each task. At the heart of MaestroMotif is the idea that decomposing a task into subtasks significantly helps decision making. MaestroMotif leverages an agent designer's intuition about a domain to identify important skills and describe them in natural language. These short descriptions then get converted into adaptable hierarchical agents through AI feedback and in-context learning. Our paper was recently published at ICLR 2025 and we open-source the whole project including the code, prompts and pre-trained models. Paper: Code: NotebookLM Podcast: This work was done with the amazing Mikael Henaff, Roberta Raileanu, Shagun Sodhani, Pascal Vincent, Amy Zhang, Pierre-Luc Bacon, Doina Precup, with equal supervision by Marlos C. Machado and Pierluca D'Oro. Take a look at the following thread:

Can AI agents adapt zero-shot, to complex multi-step language instructions in open-ended environments? We present MaestroMotif, a method for AI-assisted skill design that produces highly capable and steerable hierarchical agents. To the best of our knowledge, it is the first method that, without expert labeled datasets, solves compositional tasks requiring hundreds of steps for completion. All the modules within MaestroMotif are learned from interaction: from the highest level of planning to the lowest-level of sensorimotor control. On the open-ended domain of NetHack, it surpasses existing approaches, including those that are fine-tuned specifically for each task. At the heart of MaestroMotif is the idea that decomposing a task into subtasks significantly helps decision making. MaestroMotif leverages an agent designer's intuition about a domain to identify important skills and describe them in natural language. These short descriptions then get converted into adaptable hierarchical agents through AI feedback and in-context learning. Our paper was recently published at ICLR 2025 and we open-source the whole project including the code, prompts and pre-trained models. Paper: Code: NotebookLM Podcast: This work was done with the amazing Mikael Henaff, Roberta Raileanu, Shagun Sodhani, Pascal Vincent, Amy Zhang, Pierre-Luc Bacon, Doina Precup, with equal supervision by Marlos C. Machado and Pierluca D'Oro. Take a look at the following thread:

Martin Klissarov

80,307 views • 1 year ago

Can an inexpensive, off-the-shelf IMU be the only sensor to estimate the full state (position, velocity, orientation) of a quadrotor flying through a track at high speed and even be on-pair with vision-based localization? The answer is yes, within certain limitations! In this #RAL2023 paper, we propose a learning-based odometry algorithm that couples a model-based filter driven by the inertial measurements with a learning-based module with access to the control commands. Our system outperforms by a large margin the state-of-the-art visual-inertial odometry (#VIO) algorithms and the state-of-the-art learned-inertial odometry algorithm, #TLIO, for the task of drone racing. Additionally, we show that our system is as accurate as a VIO algorithm that uses a camera to localize to a known map of the racing track. The main limitation of our approach is that it cannot generalize to trajectories that have not been seen at training time. However, in drone racing competitions, the track is known beforehand. Human pilots spend hours or even days of practice on the race track before the competition. Similarly, our system can be trained with the data collected during practice time and deployed during the competition. Future work will investigate how to generalize to trajectories not seen at training time. The code is released! Paper: Video: Code: Kudos to Giovanni Cioffi Leonard Bauersfeld Elia Kaufmann European Research Council (ERC) University of Zurich UZH Science UZH Space Hub NCCR Robotics Aerial Core #RAL2023 #IROS2023 #SLAM

Can an inexpensive, off-the-shelf IMU be the only sensor to estimate the full state (position, velocity, orientation) of a quadrotor flying through a track at high speed and even be on-pair with vision-based localization? The answer is yes, within certain limitations! In this #RAL2023 paper, we propose a learning-based odometry algorithm that couples a model-based filter driven by the inertial measurements with a learning-based module with access to the control commands. Our system outperforms by a large margin the state-of-the-art visual-inertial odometry (#VIO) algorithms and the state-of-the-art learned-inertial odometry algorithm, #TLIO, for the task of drone racing. Additionally, we show that our system is as accurate as a VIO algorithm that uses a camera to localize to a known map of the racing track. The main limitation of our approach is that it cannot generalize to trajectories that have not been seen at training time. However, in drone racing competitions, the track is known beforehand. Human pilots spend hours or even days of practice on the race track before the competition. Similarly, our system can be trained with the data collected during practice time and deployed during the competition. Future work will investigate how to generalize to trajectories not seen at training time. The code is released! Paper: Video: Code: Kudos to Giovanni Cioffi Leonard Bauersfeld Elia Kaufmann European Research Council (ERC) University of Zurich UZH Science UZH Space Hub NCCR Robotics Aerial Core #RAL2023 #IROS2023 #SLAM

Davide Scaramuzza

37,061 views • 3 years ago

Jolt Pro is a 100x improvement over existing zkVMs. After months of researching all existing zkVMs, we realized none of them could scale to what we needed with Zero. Then we came across Justin Thaler's work and Jolt, a research project from a16z crypto. We saw massive potential in the math and, finally, a viable way to scale to our needs. So we secretly assembled a team of some of the brightest minds across cryptography, GPU programming, and ASIC design to build an internal Jolt Pro team. Jolt Pro has no precompiles; it runs only RISC-V instructions, without introducing new ad hoc, error-prone constraints. It's impossible to compare against other zkVMs because they are not proving straight RISC-V, most of their work and speed gains exist in dangerous pre-compiles. Jolt Pro scales to infinity. The number of cells you can use in parallel is only limited by the size of the datacenter. Jolt Pro has a path to 4GHz cells using the same configuration we use for our 1.61 GHz cell today. By early 2027, it will be post-quantum and set the standard for all zkVMs. We will eventually make "zero-proofs", and others can try to beat our RISC-V proving in the open. For now, we stay heads down, building Zero to be everything the industry ever wanted. And then just a little more.

Jolt Pro is a 100x improvement over existing zkVMs. After months of researching all existing zkVMs, we realized none of them could scale to what we needed with Zero. Then we came across Justin Thaler's work and Jolt, a research project from a16z crypto. We saw massive potential in the math and, finally, a viable way to scale to our needs. So we secretly assembled a team of some of the brightest minds across cryptography, GPU programming, and ASIC design to build an internal Jolt Pro team. Jolt Pro has no precompiles; it runs only RISC-V instructions, without introducing new ad hoc, error-prone constraints. It's impossible to compare against other zkVMs because they are not proving straight RISC-V, most of their work and speed gains exist in dangerous pre-compiles. Jolt Pro scales to infinity. The number of cells you can use in parallel is only limited by the size of the datacenter. Jolt Pro has a path to 4GHz cells using the same configuration we use for our 1.61 GHz cell today. By early 2027, it will be post-quantum and set the standard for all zkVMs. We will eventually make "zero-proofs", and others can try to beat our RISC-V proving in the open. For now, we stay heads down, building Zero to be everything the industry ever wanted. And then just a little more.

LayerZero

29,409 views • 7 months ago

The subject of 'owning a slave' is dense. It is something we hear a lot when we are in the FemDom Realm. Is it just fantasy? Can it actually be a lifestyle? How do we navigate this type of dynamic? How do we even get to that level of D/s? In this short clip [Exerpt from SLAVE TRAINING Part 2] I want to already bring to your attention one thing that will define if your desire for a slave (or desire as a slave) is touching more on a fantasy or... how can you actually navigate this in a realistic way. No one person 'can do it all' or should be expected to. If you want your slave to be 'the best' , assign them a specific role in which they can excel... and then build upon that. Once they 'master' your housekeeping (which takes quite a bit of real training), they can move to other levels. And an important note I want to leave here... make them EARN access to certain things in your life that sometimes you just want to delegate because you don't want to manage or don't know how to manage. Entrusting them with serious tasks that can affect your life, your business, your reputation, are on top of the ladder. Are they even qualified for the thing you want them to take off your shoulders? Start small and allow them to grow in their submission, to develop their skills and to learn how to best satisfy you without setting them up for failure by expecting too much, too quick. In the end, if you want this to truly work, you have to approach it from a place that transcends the roles. As this is consensual power exchange. And you both want to be fulfilled in that relationship.

The subject of 'owning a slave' is dense. It is something we hear a lot when we are in the FemDom Realm. Is it just fantasy? Can it actually be a lifestyle? How do we navigate this type of dynamic? How do we even get to that level of D/s? In this short clip [Exerpt from SLAVE TRAINING Part 2] I want to already bring to your attention one thing that will define if your desire for a slave (or desire as a slave) is touching more on a fantasy or... how can you actually navigate this in a realistic way. No one person 'can do it all' or should be expected to. If you want your slave to be 'the best' , assign them a specific role in which they can excel... and then build upon that. Once they 'master' your housekeeping (which takes quite a bit of real training), they can move to other levels. And an important note I want to leave here... make them EARN access to certain things in your life that sometimes you just want to delegate because you don't want to manage or don't know how to manage. Entrusting them with serious tasks that can affect your life, your business, your reputation, are on top of the ladder. Are they even qualified for the thing you want them to take off your shoulders? Start small and allow them to grow in their submission, to develop their skills and to learn how to best satisfy you without setting them up for failure by expecting too much, too quick. In the end, if you want this to truly work, you have to approach it from a place that transcends the roles. As this is consensual power exchange. And you both want to be fulfilled in that relationship.

Ms. Malissia

14,143 views • 8 months ago

.Naval: Epistemology, which is a fancy word for the theory of how knowledge grows or how knowledge growth occurs. And we've all been told since we're young that there's a scientific method and that scientists sort of do this stuff in white lab coats and we're supposed to accept it because of this thing called the scientific method. And then they give us true beliefs that we can then say, well the science is settled and we take that we move on. And we all only have a very, very vague understanding of how this works. And people say, well maybe you go out in the real world, you look at what's happening, you make all these observations, and then based on that you form a theory, you test the theory against more observations, and the more observations you get the closer you get to the truth. And once you have enough observation it's true and then you call it a scientific theory or a law and it's settled and you move on. And this is the popular conception of how science works. And as Popper pointed out and as you take even further, this is completely wrong. And so I'd love for you to get into that, which is what is knowledge? How does it grow? What is the real scientific method? And how do we figure things out? David Deutsch: I love the way you just stated the prevailing view there and laced every aspect of it with the contempt that it deserves. So you just went through touching every base. It's amazing that this series of misconceptions is still common sense. I mean, that it was common sense at a time when we didn't really have science or when science was just starting up, when the main issue in science was freeing itself from dogmatism, freeing itself from religion, freeing itself from authority, and so on. There it was understandable that people would look for an alternative source of authority and they would think, oh, it's sense impressions. We can see the world and you know, these religious people, they can't even see God and so on. And so we are confined to what we can see. That's where we get our ideas from. And as you say, that is completely false. Sense impressions, like all observation, even the most careful scientific observation is all theory laden. And theories are inherently fallible. I mean, we actually want to replace our best theories. Everybody who does a PhD is technically anyway, working to overturn something in the existing body of knowledge. You're not turned away at the door if you say, I don't believe this stuff, I'm going to produce something better. Whereas for most of human history, that was exactly what you were forbidden to do. The idea was that we already had all the important knowledge. If you want to discover something new, what you had to make sure of was that it didn't contradict the existing knowledge. Now, you have to make sure that it does contradict existing knowledge. So more or less. Naval: Yeah, it's this tradition of criticism that you've talked about in the West, that the Enlightenment really ushered in the Enlightenment era. David Deutsch: It has been institutionalized. So in many ways, our institutions are wiser than we are. So the institutions of science, for instance, have this built in, even if scientists actually don't always act that way. In fact, they often don't act that way, and act in a dogmatic way and try to preserve the status quo and are resistant to new ideas and so on. But the institutions, the way the procedures of science work, makes the right thing happen in the end anyway, regardless of what the people are trying to do. Naval: So you're saying the knowledge of the true scientific method is embedded in the institutions of science in the PhD process? David Deutsch: Well, the best scientific method that we know of, and one shouldn't really think of it as a method, you know, there's this wonderful lecture by Popper when he first was made a professor at the London School of Economics. He was made a professor of scientific method, and his first six lectures, I wish the rest of them were, the first six lectures are on the internet somewhere. And he starts the first one by saying, I am the first professor of scientific method in the British Empire. The British Empire still existed at the time, more or less. And so the first thing I want to say to you is that there is no such thing as the scientific method. And then he goes on from there. So this subject does not exist. So if any of you have come here to learn the handle that you have to turn in order to make scientific knowledge come out the other end, you're going to be disappointed.

.Naval: Epistemology, which is a fancy word for the theory of how knowledge grows or how knowledge growth occurs. And we've all been told since we're young that there's a scientific method and that scientists sort of do this stuff in white lab coats and we're supposed to accept it because of this thing called the scientific method. And then they give us true beliefs that we can then say, well the science is settled and we take that we move on. And we all only have a very, very vague understanding of how this works. And people say, well maybe you go out in the real world, you look at what's happening, you make all these observations, and then based on that you form a theory, you test the theory against more observations, and the more observations you get the closer you get to the truth. And once you have enough observation it's true and then you call it a scientific theory or a law and it's settled and you move on. And this is the popular conception of how science works. And as Popper pointed out and as you take even further, this is completely wrong. And so I'd love for you to get into that, which is what is knowledge? How does it grow? What is the real scientific method? And how do we figure things out? David Deutsch: I love the way you just stated the prevailing view there and laced every aspect of it with the contempt that it deserves. So you just went through touching every base. It's amazing that this series of misconceptions is still common sense. I mean, that it was common sense at a time when we didn't really have science or when science was just starting up, when the main issue in science was freeing itself from dogmatism, freeing itself from religion, freeing itself from authority, and so on. There it was understandable that people would look for an alternative source of authority and they would think, oh, it's sense impressions. We can see the world and you know, these religious people, they can't even see God and so on. And so we are confined to what we can see. That's where we get our ideas from. And as you say, that is completely false. Sense impressions, like all observation, even the most careful scientific observation is all theory laden. And theories are inherently fallible. I mean, we actually want to replace our best theories. Everybody who does a PhD is technically anyway, working to overturn something in the existing body of knowledge. You're not turned away at the door if you say, I don't believe this stuff, I'm going to produce something better. Whereas for most of human history, that was exactly what you were forbidden to do. The idea was that we already had all the important knowledge. If you want to discover something new, what you had to make sure of was that it didn't contradict the existing knowledge. Now, you have to make sure that it does contradict existing knowledge. So more or less. Naval: Yeah, it's this tradition of criticism that you've talked about in the West, that the Enlightenment really ushered in the Enlightenment era. David Deutsch: It has been institutionalized. So in many ways, our institutions are wiser than we are. So the institutions of science, for instance, have this built in, even if scientists actually don't always act that way. In fact, they often don't act that way, and act in a dogmatic way and try to preserve the status quo and are resistant to new ideas and so on. But the institutions, the way the procedures of science work, makes the right thing happen in the end anyway, regardless of what the people are trying to do. Naval: So you're saying the knowledge of the true scientific method is embedded in the institutions of science in the PhD process? David Deutsch: Well, the best scientific method that we know of, and one shouldn't really think of it as a method, you know, there's this wonderful lecture by Popper when he first was made a professor at the London School of Economics. He was made a professor of scientific method, and his first six lectures, I wish the rest of them were, the first six lectures are on the internet somewhere. And he starts the first one by saying, I am the first professor of scientific method in the British Empire. The British Empire still existed at the time, more or less. And so the first thing I want to say to you is that there is no such thing as the scientific method. And then he goes on from there. So this subject does not exist. So if any of you have come here to learn the handle that you have to turn in order to make scientific knowledge come out the other end, you're going to be disappointed.

Deutsch Explains

116,352 views • 1 year ago

Jev is cool. So is it's OSS companion, Laya. The Latest Cool Thing In AI™ tends to get a lot of hype, sometimes without everyone even understanding it. So... what is this thing? Jev is an AI model that consumes input and produces output VERY differently than chat, claude, grok. The input is two things: 1) Text state to assess. Email, html, code, whatever. 2) A set of questions which will be asked about the attached state. The canonical example from TypeSafe's docs is to identify the urgency of a support ticket. We pass the model the customer text + a single noul question "is this urgent?". Jev returns a full set of JSON. This JSON is not generated with token-by-token autoregression. Jev is not trained to produce sequences of text tokens, rather to answer questions, and guarantees well-formed responses. In the example below, we see it produces a 0.99 probability (on a 0-1.0 scale) that the answer is "yes." Jev supports exactly three types of questions (seconds example in video): a) Noul: 0–1 probability that the answer to a yes/no question is "yes." b) Choice: Ask question with pre-defined set of answers. Jev chooses the best and assigns probabilities to each. c) Score: Ask question with pre-defined scale of answers. Jev produces a position on the scale. Jev computes answers for all questions in parallel, making responses super fast even for many questions in a single request. This might seem like a narrow set of capabilities, but in the right contexts leads to incredible potential. It also makes for a useful API / primitive for programming, since the outputs are... *ahem*... type-safe and predictable in structure. Jev is not going to replace LLMs for writing your code, auto-generating your docs, or being at the core of an agent harness. But Jev IS incredibly cool, and will be used to build a lot of amazing tech. Hope this helps.

Jev is cool. So is it's OSS companion, Laya. The Latest Cool Thing In AI™ tends to get a lot of hype, sometimes without everyone even understanding it. So... what is this thing? Jev is an AI model that consumes input and produces output VERY differently than chat, claude, grok. The input is two things: 1) Text state to assess. Email, html, code, whatever. 2) A set of questions which will be asked about the attached state. The canonical example from TypeSafe's docs is to identify the urgency of a support ticket. We pass the model the customer text + a single noul question "is this urgent?". Jev returns a full set of JSON. This JSON is not generated with token-by-token autoregression. Jev is not trained to produce sequences of text tokens, rather to answer questions, and guarantees well-formed responses. In the example below, we see it produces a 0.99 probability (on a 0-1.0 scale) that the answer is "yes." Jev supports exactly three types of questions (seconds example in video): a) Noul: 0–1 probability that the answer to a yes/no question is "yes." b) Choice: Ask question with pre-defined set of answers. Jev chooses the best and assigns probabilities to each. c) Score: Ask question with pre-defined scale of answers. Jev produces a position on the scale. Jev computes answers for all questions in parallel, making responses super fast even for many questions in a single request. This might seem like a narrow set of capabilities, but in the right contexts leads to incredible potential. It also makes for a useful API / primitive for programming, since the outputs are... ahem... type-safe and predictable in structure. Jev is not going to replace LLMs for writing your code, auto-generating your docs, or being at the core of an agent harness. But Jev IS incredibly cool, and will be used to build a lot of amazing tech. Hope this helps.

Ben Dicken

40,810 views • 29 days ago