Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

🔍 Introducing BackSearch. LLMs are increasingly asked to predict the future, but a good backtest requires a snapshot of the internet at a point in time. BackSearch allows LLMs to search the web as it was on a particular date. It’s great for: 🔮 Forecasting and prediction markets. 📈... show more

General Reasoning

6,788 subscribers

419,098 views • 1 month ago •via X (Twitter)

Science & Technology Education Finance

Anya Rossi• Live Now

Private livecam show

35 Comments

will brown1 month ago

oh hell yeah this is gonna print

General Reasoning1 month ago

Details on how to try BackSearch out here:

Daniel Lougen1 month ago

Wayback machine for AI, fuck thats smart. I wonder what kind of training data we might get out of this

Sam Z Liu1 month ago

but upgraded?

Cruncher Jean1 month ago

You will still inject lookahead bias from the model if it's not completely retrain from scratch. Good for basic RL env but bad for rigorous backtesting. Check ChronosLLM:

Sriraam1 month ago

We were just talking about this yesterday lol 🔥🔥

Josh Harris1 month ago

This is exactly what we need when we were building our frontier financial judgement benchmark - will try to integrate for v2 :)

Vivek Iyer1 month ago

best data product released in a minute damn

jakedineenasu1 month ago

This is what we target in our paper "Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters", although we look at frozen slices of Reddit as a proxy for general markets against replayed Polymarket events.

Muratcan Koylan1 month ago

The timing is perfect; I really needed this.

Albert Catalán Tatjer1 month ago

I was just looking for something like this

M1 month ago

watch out ppl are gonna do so much prime-rl on this

λux1 month ago

this is super cool!! 🙌🏻

Anthony Tamasi1 month ago

This is a great idea, but models are pretrained on info up to some cutoff date. Wouldn’t there still be data leakage from future events even without tool use?

prady1 month ago

Woah!

Alex Peng1 month ago

This is so cool

Robin Salimans1 month ago

this looks super useful

Anirudh Ravichandran1 month ago

excellent, I can totally see the hill climbing with reproducible reward on search environments going brrr @willcb

Mani 🪽1 month ago

this is super cool !!!!

Vantix AI Agency1 month ago

Searching the web from any past date changes how models get tested

Shreyas Pimpalgaonkar1 month ago

Wow, this is actually so useful! Congratulations

Nayeem1 month ago

Incredible

Thomas Mattimore 🇺🇸1 month ago

Will be a cool way to track various predictions made over time

Restitutor1 month ago

Awesome, I can think of other uses for this

Oscar Hong1 month ago

this is genius. always wondered why there wasn’t a benchmark for predictions. does this only work for events that happened after a model’s training date?

Matan Halevy1 month ago

unreal, we’re working on a benchmark this would be useful for. Excited to give it a try

Luigi Pagani1 month ago

Wow, so cool!

Jia Ming (إحسان)1 month ago

Google should be doing this

callum1 month ago

super cool! and very useful

आशीष खरे1 month ago

This is new. I like the general idea. Great one!

Yusuf1 month ago

Need this

Ha Hoang1 month ago

is this way back machine but for AI agent?

ethan1 month ago

great idea

2cents1 month ago

@teortaxesTex This is very useful utility

BowTiedStingray1 month ago

Beautiful - thank you for this

Related Videos

We built a time machine for the web. Introducing Exa Snapshot: an index of 400 billion historical snapshots of webpages that lets you search as if it's the past. Snapshot is already being used for backtesting prediction models, RL at labs, exploring the pre-AI web, and more.

We built a time machine for the web. Introducing Exa Snapshot: an index of 400 billion historical snapshots of webpages that lets you search as if it's the past. Snapshot is already being used for backtesting prediction models, RL at labs, exploring the pre-AI web, and more.

Exa

662,939 views • 1 day ago

Introducing Mintlify Index: the first publisher-direct search layer for technical knowledge, built on Mintlify’s ecosystem of more than 200,000 orgs. In our benchmark against Context7, agents using Mintlify Index finished tasks at twice the speed, and with 9% higher factual accuracy. Today, we're opening up the Index MCP server for everyone to try, fully for free:

Introducing Mintlify Index: the first publisher-direct search layer for technical knowledge, built on Mintlify’s ecosystem of more than 200,000 orgs. In our benchmark against Context7, agents using Mintlify Index finished tasks at twice the speed, and with 9% higher factual accuracy. Today, we're opening up the Index MCP server for everyone to try, fully for free:

Mintlify

150,593 views • 1 month ago

The network for machine intelligence Two years ago, we laid out our vision for a machine learning compute protocol. One that connects every device in the world into an open network for machine intelligence, with no gatekeepers or artificial boundaries. This week, we’ll be sharing some of our early progress, beginning with RL Swarm, a peer-to-peer system for collaborative reinforcement learning over the internet. Next month, we’ll open our Testnet, allowing anyone to contribute to the frontier of open machine intelligence. Introducing RL Swarm RL Swarm is a fully open source system for collaborative reinforcement learning over the internet. It is a live demo of our research findings, which show that models training with RL learn faster when they train as a collective swarm than they do on their own. Join our swarm now to see this in practice. You can participate with consumer hardware at home or a powerful GPU in the cloud. You can follow along with the swarm’s progress by following the links below.

The network for machine intelligence Two years ago, we laid out our vision for a machine learning compute protocol. One that connects every device in the world into an open network for machine intelligence, with no gatekeepers or artificial boundaries. This week, we’ll be sharing some of our early progress, beginning with RL Swarm, a peer-to-peer system for collaborative reinforcement learning over the internet. Next month, we’ll open our Testnet, allowing anyone to contribute to the frontier of open machine intelligence. Introducing RL Swarm RL Swarm is a fully open source system for collaborative reinforcement learning over the internet. It is a live demo of our research findings, which show that models training with RL learn faster when they train as a collective swarm than they do on their own. Join our swarm now to see this in practice. You can participate with consumer hardware at home or a powerful GPU in the cloud. You can follow along with the swarm’s progress by following the links below.

gensyn

228,919 views • 1 year ago

Introducing a new look, for a new world. At Verse, we’re on a mission to redefine digital environments. We aim to bridge the gap between the physical and digital worlds by providing our users with an experience that closely mimics real-life interactions. This is a world built for players, brands and communities to thrive in a new era of experiential gameplay and engagement. Welcome to the Internet of Reality. Welcome to Verse World.

Introducing a new look, for a new world. At Verse, we’re on a mission to redefine digital environments. We aim to bridge the gap between the physical and digital worlds by providing our users with an experience that closely mimics real-life interactions. This is a world built for players, brands and communities to thrive in a new era of experiential gameplay and engagement. Welcome to the Internet of Reality. Welcome to Verse World.

Verse World

64,590 views • 1 year ago

🗣Jonas Vingegaard: "It was a stressful day, but a good day for us. We ended up in the first group, That was our plan, to try to go in the wind with 20km to go. The team kept me out of trouble at any point, thanks a lot to the team."

🗣Jonas Vingegaard: "It was a stressful day, but a good day for us. We ended up in the first group, That was our plan, to try to go in the wind with 20km to go. The team kept me out of trouble at any point, thanks a lot to the team."

#TeamVingegaard

46,641 views • 1 year ago

Based Prediction Markets In partnership with Polymarket, the Based team will be launching prediction market access to users of Based. This manifests as a highly performant and intuitive way for users to trade prediction markets across iOS, Android, and our Web terminal product. Utilising Polymarket's deep liquidity and robust settlement, we carve out a new market segment for our users to trade. This is built in deep integration with Hyperliquid's hypercore to allow users to easily move funds in and out of their Perpetual accounts into Prediction Market account. The BasedPals will also receive an unique utility in-line with our agentic trading roadmap in partnership with Polymarket. With the integration of Prediction Market access, we diversify and increase Based's revenue streams as we move ahead into our eventual TGE. In order to grow our Prediction Market business, the Based team will be setting aside 2mil Based Gold every epoch (starting #14) for Based users who trade on our Prediction Markets. Stay Based.

Based Prediction Markets In partnership with Polymarket, the Based team will be launching prediction market access to users of Based. This manifests as a highly performant and intuitive way for users to trade prediction markets across iOS, Android, and our Web terminal product. Utilising Polymarket's deep liquidity and robust settlement, we carve out a new market segment for our users to trade. This is built in deep integration with Hyperliquid's hypercore to allow users to easily move funds in and out of their Perpetual accounts into Prediction Market account. The BasedPals will also receive an unique utility in-line with our agentic trading roadmap in partnership with Polymarket. With the integration of Prediction Market access, we diversify and increase Based's revenue streams as we move ahead into our eventual TGE. In order to grow our Prediction Market business, the Based team will be setting aside 2mil Based Gold every epoch (starting #14) for Based users who trade on our Prediction Markets. Stay Based.

Based

230,943 views • 10 months ago

Many have said that the differences and divisions in our world are too great for us to come together for the common good. But the Summit of the Future showed that with a spirit of dialogue and compromise, we can join forces to steer our world to a more sustainable path. As this year’s #UNGA activities wind down, we must remember that this is not the end. It is in all our interests to choose the future we want and to guide our world towards it. The journey continues. Let’s keep going.

Many have said that the differences and divisions in our world are too great for us to come together for the common good. But the Summit of the Future showed that with a spirit of dialogue and compromise, we can join forces to steer our world to a more sustainable path. As this year’s #UNGA activities wind down, we must remember that this is not the end. It is in all our interests to choose the future we want and to guide our world towards it. The journey continues. Let’s keep going.

António Guterres

110,811 views • 1 year ago

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

Xiang Yue

56,696 views • 2 years ago

Hiring RL Engineer! Started off as a curious project at Lossfunk to push the boundaries of LLMs in social reasoning - we are now building RL environments, data, and benchmarks to simulate more real-world scenarios. If you want to train SoTA RL models over multi-GPUs (H200s/B200s) to unlock next AI frontier, this is for you.

Hiring RL Engineer! Started off as a curious project at Lossfunk to push the boundaries of LLMs in social reasoning - we are now building RL environments, data, and benchmarks to simulate more real-world scenarios. If you want to train SoTA RL models over multi-GPUs (H200s/B200s) to unlock next AI frontier, this is for you.

Satpal Singh Rathore

46,010 views • 1 year ago

🌍 We’re at a Turning Point in History The future of robotics and human language hinges on teaching machines to inhabit our world. Nils and @builtnpublicpod point out that for decades, humans entered digital spaces for experiences. This decade, it’s reversing. We’re inviting the digital into our world. 🤖✨ This shift isn’t just technological, it’s holy. It coincides with DePIN, a permanent software upgrade to capitalism, creating the infrastructure AI needs to perceive the world. Follow @Auki is bringing this change now. 🚀

🌍 We’re at a Turning Point in History The future of robotics and human language hinges on teaching machines to inhabit our world. Nils and @builtnpublicpod point out that for decades, humans entered digital spaces for experiences. This decade, it’s reversing. We’re inviting the digital into our world. 🤖✨ This shift isn’t just technological, it’s holy. It coincides with DePIN, a permanent software upgrade to capitalism, creating the infrastructure AI needs to perceive the world. Follow @Auki is bringing this change now. 🚀

Santeri Aramo

10,195 views • 1 year ago

Check out what some of our participants had to say about this nostalgic reunion with their childhood game at the first limited-time public test of Classic World. A big thank you to everyone that took the time to stop by and left valuable feedback for us to shape the future of the game! Sign up now for a chance at participating in our MapleStory Classic World Closed Online Test and experience the official faithful recreation! Link Below ⬇️

Check out what some of our participants had to say about this nostalgic reunion with their childhood game at the first limited-time public test of Classic World. A big thank you to everyone that took the time to stop by and left valuable feedback for us to shape the future of the game! Sign up now for a chance at participating in our MapleStory Classic World Closed Online Test and experience the official faithful recreation! Link Below ⬇️

MapleStory 🍁

11,632 views • 5 months ago

Introducing RAGs v3 🌟: Build a RAG bot that can also search the web 🌐, to find answers that aren’t immediately in its corpus. Do this all in natural language, not code! 💬 Get a better experience than ChatGPT + Bing ⚡️ We used our integration with Metaphor - a search engine designed for LLMs - to pull in relevant text from the internet. Additionally: - you can now view the tools that the agent is using. - note: web search is only accessible for our OpenAI agent Check out the video below, and get started with the repo today. Repo:

Introducing RAGs v3 🌟: Build a RAG bot that can also search the web 🌐, to find answers that aren’t immediately in its corpus. Do this all in natural language, not code! 💬 Get a better experience than ChatGPT + Bing ⚡️ We used our integration with Metaphor - a search engine designed for LLMs - to pull in relevant text from the internet. Additionally: - you can now view the tools that the agent is using. - note: web search is only accessible for our OpenAI agent Check out the video below, and get started with the repo today. Repo:

LlamaIndex 🦙

76,371 views • 2 years ago

Friends, this Sunday, right before the commencement of Lent, the Church is giving us something of great moment to reflect on—namely, the centrality of freedom and choice for the good at the center of the spiritual life. As Thomas More puts it in “A Man for All Seasons,” “God made animals for innocence and plants for their simplicity. But Man He made to serve Him wittily, in the tangle of his mind.” God wants us to give him glory in a particular way: through our intellect and will—our search for truth and our love for him.

Friends, this Sunday, right before the commencement of Lent, the Church is giving us something of great moment to reflect on—namely, the centrality of freedom and choice for the good at the center of the spiritual life. As Thomas More puts it in “A Man for All Seasons,” “God made animals for innocence and plants for their simplicity. But Man He made to serve Him wittily, in the tangle of his mind.” God wants us to give him glory in a particular way: through our intellect and will—our search for truth and our love for him.

Bishop Robert Barron

44,475 views • 7 months ago

.SnowflakeDB is thrilled to announce #SnowflakeArctic: A state-of-the-art large language model uniquely designed to be the most open, enterprise-grade LLM on the market. This is a big step forward for open source LLMs. And it’s a big moment for Snowflake in our #AI journey as we continue to build best-in-class enterprise-grade products for our customers. The era of enterprise AI is here. 🚀

.SnowflakeDB is thrilled to announce #SnowflakeArctic: A state-of-the-art large language model uniquely designed to be the most open, enterprise-grade LLM on the market. This is a big step forward for open source LLMs. And it’s a big moment for Snowflake in our #AI journey as we continue to build best-in-class enterprise-grade products for our customers. The era of enterprise AI is here. 🚀

sridhar

243,833 views • 2 years ago

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.

Jing Yu Koh

124,894 views • 2 years ago

We’re thrilled to announce that 2 of our projects are nominated for E-Commerce Site of the Year on awwwards. and that our team is nominated for Studio of the Year for the second time in a row. Take a look at our reel to get a glimpse of the nominated projects & last year’s work!

We’re thrilled to announce that 2 of our projects are nominated for E-Commerce Site of the Year on awwwards. and that our team is nominated for Studio of the Year for the second time in a row. Take a look at our reel to get a glimpse of the nominated projects & last year’s work!

Antinomy Studio

18,451 views • 3 years ago

With the UK on course to become the largest contributor of e-waste in the world, we’re highlighting how consumers can tackle it this #EarthDay 🌍 From our O2 Recycle scheme, to our Community Calling initiative, in partnership with Hubbub, we’re supporting the #sustainable disposable of unwanted devices – to avoid them ending up in landfill ♻️ Find out more about our new Virgin TV educational content for #EarthDay2024: Take a look at our commitment to operating in a more sustainable way with our Better Connections Plan:

With the UK on course to become the largest contributor of e-waste in the world, we’re highlighting how consumers can tackle it this #EarthDay 🌍 From our O2 Recycle scheme, to our Community Calling initiative, in partnership with Hubbub, we’re supporting the #sustainable disposable of unwanted devices – to avoid them ending up in landfill ♻️ Find out more about our new Virgin TV educational content for #EarthDay2024: Take a look at our commitment to operating in a more sustainable way with our Better Connections Plan:

Virgin Media O2 News

15,056 views • 2 years ago

Insider trading is a feature of Polymarket. A Google employee making $1M+ on search trends is simply efficient price discovery. Platform does not ask if your advantage is fair. It only asks if you are right But this logic leads to a dark place: Assassination Markets. If a market allows bets on a death, it creates a direct financial incentive to... resolve the market When millions are on the line, the platform stops being a passive mirror of the future and becomes an engine for shaping it Are Prediction Markets the most honest and cynical technology of our time?

Insider trading is a feature of Polymarket. A Google employee making $1M+ on search trends is simply efficient price discovery. Platform does not ask if your advantage is fair. It only asks if you are right But this logic leads to a dark place: Assassination Markets. If a market allows bets on a death, it creates a direct financial incentive to... resolve the market When millions are on the line, the platform stops being a passive mirror of the future and becomes an engine for shaping it Are Prediction Markets the most honest and cynical technology of our time?

Lirratø

212,694 views • 9 months ago