正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

🔍 Introducing BackSearch. LLMs are increasingly asked to predict the future, but a good backtest requires a snapshot of the internet at a point in time. BackSearch allows LLMs to search the web as it was on a particular date. It’s great for: 🔮 Forecasting and prediction markets. 📈... show more

General Reasoning

6,788 subscribers

419,098 次观看 • 1 个月前 •via X (Twitter)

科学技术教育财经

Anya Rossi• Live Now

Private livecam show

35 条评论

will brown 的头像

will brown1 个月前

oh hell yeah this is gonna print

General Reasoning 的头像

General Reasoning1 个月前

Details on how to try BackSearch out here:

Daniel Lougen 的头像

Daniel Lougen1 个月前

Wayback machine for AI, fuck thats smart. I wonder what kind of training data we might get out of this

Sam Z Liu 的头像

Sam Z Liu1 个月前

but upgraded?

Cruncher Jean 的头像

Cruncher Jean1 个月前

You will still inject lookahead bias from the model if it's not completely retrain from scratch. Good for basic RL env but bad for rigorous backtesting. Check ChronosLLM:

Sriraam 的头像

Sriraam1 个月前

We were just talking about this yesterday lol 🔥🔥

Josh Harris 的头像

Josh Harris1 个月前

This is exactly what we need when we were building our frontier financial judgement benchmark - will try to integrate for v2 :)

Vivek Iyer 的头像

Vivek Iyer1 个月前

best data product released in a minute damn

jakedineenasu 的头像

jakedineenasu1 个月前

This is what we target in our paper "Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters", although we look at frozen slices of Reddit as a proxy for general markets against replayed Polymarket events.

Muratcan Koylan 的头像

Muratcan Koylan1 个月前

The timing is perfect; I really needed this.

Albert Catalán Tatjer 的头像

Albert Catalán Tatjer1 个月前

I was just looking for something like this

M 的头像

M1 个月前

watch out ppl are gonna do so much prime-rl on this

λux 的头像

λux1 个月前

this is super cool!! 🙌🏻

Anthony Tamasi 的头像

Anthony Tamasi1 个月前

This is a great idea, but models are pretrained on info up to some cutoff date. Wouldn’t there still be data leakage from future events even without tool use?

prady 的头像

prady1 个月前

Woah!

Alex Peng 的头像

Alex Peng1 个月前

This is so cool

Robin Salimans 的头像

Robin Salimans1 个月前

this looks super useful

Anirudh Ravichandran 的头像

Anirudh Ravichandran1 个月前

excellent, I can totally see the hill climbing with reproducible reward on search environments going brrr @willcb

Mani 🪽 的头像

Mani 🪽1 个月前

this is super cool !!!!

Vantix AI Agency 的头像

Vantix AI Agency1 个月前

Searching the web from any past date changes how models get tested

Shreyas Pimpalgaonkar 的头像

Shreyas Pimpalgaonkar1 个月前

Wow, this is actually so useful! Congratulations

Nayeem 的头像

Nayeem1 个月前

Incredible

Thomas Mattimore 🇺🇸 的头像

Thomas Mattimore 🇺🇸1 个月前

Will be a cool way to track various predictions made over time

Restitutor 的头像

Restitutor1 个月前

Awesome, I can think of other uses for this

Oscar Hong 的头像

Oscar Hong1 个月前

this is genius. always wondered why there wasn’t a benchmark for predictions. does this only work for events that happened after a model’s training date?

Matan Halevy 的头像

Matan Halevy1 个月前

unreal, we’re working on a benchmark this would be useful for. Excited to give it a try

Luigi Pagani 的头像

Luigi Pagani1 个月前

Wow, so cool!

Jia Ming (إحسان) 的头像

Jia Ming (إحسان)1 个月前

Google should be doing this

callum 的头像

callum1 个月前

super cool! and very useful

आशीष खरे 的头像

आशीष खरे1 个月前

This is new. I like the general idea. Great one!

Yusuf 的头像

Yusuf1 个月前

Need this

Ha Hoang 的头像

Ha Hoang1 个月前

is this way back machine but for AI agent?

ethan 的头像

ethan1 个月前

great idea

2cents 的头像

2cents1 个月前

@teortaxesTex This is very useful utility

BowTiedStingray 的头像

BowTiedStingray1 个月前

Beautiful - thank you for this

相关视频

We built a time machine for the web. Introducing Exa Snapshot: an index of 400 billion historical snapshots of webpages that lets you search as if it's the past. Snapshot is already being used for backtesting prediction models, RL at labs, exploring the pre-AI web, and more.

We built a time machine for the web. Introducing Exa Snapshot: an index of 400 billion historical snapshots of webpages that lets you search as if it's the past. Snapshot is already being used for backtesting prediction models, RL at labs, exploring the pre-AI web, and more.

Exa

662,939 次观看 • 1 天前

Introducing Mintlify Index: the first publisher-direct search layer for technical knowledge, built on Mintlify’s ecosystem of more than 200,000 orgs. In our benchmark against Context7, agents using Mintlify Index finished tasks at twice the speed, and with 9% higher factual accuracy. Today, we're opening up the Index MCP server for everyone to try, fully for free:

Introducing Mintlify Index: the first publisher-direct search layer for technical knowledge, built on Mintlify’s ecosystem of more than 200,000 orgs. In our benchmark against Context7, agents using Mintlify Index finished tasks at twice the speed, and with 9% higher factual accuracy. Today, we're opening up the Index MCP server for everyone to try, fully for free:

Mintlify

150,593 次观看 • 1 个月前

The network for machine intelligence Two years ago, we laid out our vision for a machine learning compute protocol. One that connects every device in the world into an open network for machine intelligence, with no gatekeepers or artificial boundaries. This week, we’ll be sharing some of our early progress, beginning with RL Swarm, a peer-to-peer system for collaborative reinforcement learning over the internet. Next month, we’ll open our Testnet, allowing anyone to contribute to the frontier of open machine intelligence. Introducing RL Swarm RL Swarm is a fully open source system for collaborative reinforcement learning over the internet. It is a live demo of our research findings, which show that models training with RL learn faster when they train as a collective swarm than they do on their own. Join our swarm now to see this in practice. You can participate with consumer hardware at home or a powerful GPU in the cloud. You can follow along with the swarm’s progress by following the links below.

The network for machine intelligence Two years ago, we laid out our vision for a machine learning compute protocol. One that connects every device in the world into an open network for machine intelligence, with no gatekeepers or artificial boundaries. This week, we’ll be sharing some of our early progress, beginning with RL Swarm, a peer-to-peer system for collaborative reinforcement learning over the internet. Next month, we’ll open our Testnet, allowing anyone to contribute to the frontier of open machine intelligence. Introducing RL Swarm RL Swarm is a fully open source system for collaborative reinforcement learning over the internet. It is a live demo of our research findings, which show that models training with RL learn faster when they train as a collective swarm than they do on their own. Join our swarm now to see this in practice. You can participate with consumer hardware at home or a powerful GPU in the cloud. You can follow along with the swarm’s progress by following the links below.

gensyn

228,919 次观看 • 1 年前

Introducing a new look, for a new world. At Verse, we’re on a mission to redefine digital environments. We aim to bridge the gap between the physical and digital worlds by providing our users with an experience that closely mimics real-life interactions. This is a world built for players, brands and communities to thrive in a new era of experiential gameplay and engagement. Welcome to the Internet of Reality. Welcome to Verse World.

Introducing a new look, for a new world. At Verse, we’re on a mission to redefine digital environments. We aim to bridge the gap between the physical and digital worlds by providing our users with an experience that closely mimics real-life interactions. This is a world built for players, brands and communities to thrive in a new era of experiential gameplay and engagement. Welcome to the Internet of Reality. Welcome to Verse World.

Verse World

64,590 次观看 • 1 年前

🗣Jonas Vingegaard: "It was a stressful day, but a good day for us. We ended up in the first group, That was our plan, to try to go in the wind with 20km to go. The team kept me out of trouble at any point, thanks a lot to the team."

🗣Jonas Vingegaard: "It was a stressful day, but a good day for us. We ended up in the first group, That was our plan, to try to go in the wind with 20km to go. The team kept me out of trouble at any point, thanks a lot to the team."

#TeamVingegaard

46,641 次观看 • 1 年前

Based Prediction Markets In partnership with Polymarket, the Based team will be launching prediction market access to users of Based. This manifests as a highly performant and intuitive way for users to trade prediction markets across iOS, Android, and our Web terminal product. Utilising Polymarket's deep liquidity and robust settlement, we carve out a new market segment for our users to trade. This is built in deep integration with Hyperliquid's hypercore to allow users to easily move funds in and out of their Perpetual accounts into Prediction Market account. The BasedPals will also receive an unique utility in-line with our agentic trading roadmap in partnership with Polymarket. With the integration of Prediction Market access, we diversify and increase Based's revenue streams as we move ahead into our eventual TGE. In order to grow our Prediction Market business, the Based team will be setting aside 2mil Based Gold every epoch (starting #14) for Based users who trade on our Prediction Markets. Stay Based.

Based Prediction Markets In partnership with Polymarket, the Based team will be launching prediction market access to users of Based. This manifests as a highly performant and intuitive way for users to trade prediction markets across iOS, Android, and our Web terminal product. Utilising Polymarket's deep liquidity and robust settlement, we carve out a new market segment for our users to trade. This is built in deep integration with Hyperliquid's hypercore to allow users to easily move funds in and out of their Perpetual accounts into Prediction Market account. The BasedPals will also receive an unique utility in-line with our agentic trading roadmap in partnership with Polymarket. With the integration of Prediction Market access, we diversify and increase Based's revenue streams as we move ahead into our eventual TGE. In order to grow our Prediction Market business, the Based team will be setting aside 2mil Based Gold every epoch (starting #14) for Based users who trade on our Prediction Markets. Stay Based.

Based

230,943 次观看 • 10 个月前

Many have said that the differences and divisions in our world are too great for us to come together for the common good. But the Summit of the Future showed that with a spirit of dialogue and compromise, we can join forces to steer our world to a more sustainable path. As this year’s #UNGA activities wind down, we must remember that this is not the end. It is in all our interests to choose the future we want and to guide our world towards it. The journey continues. Let’s keep going.

Many have said that the differences and divisions in our world are too great for us to come together for the common good. But the Summit of the Future showed that with a spirit of dialogue and compromise, we can join forces to steer our world to a more sustainable path. As this year’s #UNGA activities wind down, we must remember that this is not the end. It is in all our interests to choose the future we want and to guide our world towards it. The journey continues. Let’s keep going.

António Guterres

110,811 次观看 • 1 年前

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

Xiang Yue

56,696 次观看 • 2 年前

Hiring RL Engineer! Started off as a curious project at Lossfunk to push the boundaries of LLMs in social reasoning - we are now building RL environments, data, and benchmarks to simulate more real-world scenarios. If you want to train SoTA RL models over multi-GPUs (H200s/B200s) to unlock next AI frontier, this is for you.

Hiring RL Engineer! Started off as a curious project at Lossfunk to push the boundaries of LLMs in social reasoning - we are now building RL environments, data, and benchmarks to simulate more real-world scenarios. If you want to train SoTA RL models over multi-GPUs (H200s/B200s) to unlock next AI frontier, this is for you.

Satpal Singh Rathore

46,010 次观看 • 1 年前

🌍 We’re at a Turning Point in History The future of robotics and human language hinges on teaching machines to inhabit our world. Nils and @builtnpublicpod point out that for decades, humans entered digital spaces for experiences. This decade, it’s reversing. We’re inviting the digital into our world. 🤖✨ This shift isn’t just technological, it’s holy. It coincides with DePIN, a permanent software upgrade to capitalism, creating the infrastructure AI needs to perceive the world. Follow @Auki is bringing this change now. 🚀

🌍 We’re at a Turning Point in History The future of robotics and human language hinges on teaching machines to inhabit our world. Nils and @builtnpublicpod point out that for decades, humans entered digital spaces for experiences. This decade, it’s reversing. We’re inviting the digital into our world. 🤖✨ This shift isn’t just technological, it’s holy. It coincides with DePIN, a permanent software upgrade to capitalism, creating the infrastructure AI needs to perceive the world. Follow @Auki is bringing this change now. 🚀

Santeri Aramo

10,195 次观看 • 1 年前

Check out what some of our participants had to say about this nostalgic reunion with their childhood game at the first limited-time public test of Classic World. A big thank you to everyone that took the time to stop by and left valuable feedback for us to shape the future of the game! Sign up now for a chance at participating in our MapleStory Classic World Closed Online Test and experience the official faithful recreation! Link Below ⬇️

Check out what some of our participants had to say about this nostalgic reunion with their childhood game at the first limited-time public test of Classic World. A big thank you to everyone that took the time to stop by and left valuable feedback for us to shape the future of the game! Sign up now for a chance at participating in our MapleStory Classic World Closed Online Test and experience the official faithful recreation! Link Below ⬇️

MapleStory 🍁

11,632 次观看 • 5 个月前

Introducing RAGs v3 🌟: Build a RAG bot that can also search the web 🌐, to find answers that aren’t immediately in its corpus. Do this all in natural language, not code! 💬 Get a better experience than ChatGPT + Bing ⚡️ We used our integration with Metaphor - a search engine designed for LLMs - to pull in relevant text from the internet. Additionally: - you can now view the tools that the agent is using. - note: web search is only accessible for our OpenAI agent Check out the video below, and get started with the repo today. Repo:

Introducing RAGs v3 🌟: Build a RAG bot that can also search the web 🌐, to find answers that aren’t immediately in its corpus. Do this all in natural language, not code! 💬 Get a better experience than ChatGPT + Bing ⚡️ We used our integration with Metaphor - a search engine designed for LLMs - to pull in relevant text from the internet. Additionally: - you can now view the tools that the agent is using. - note: web search is only accessible for our OpenAI agent Check out the video below, and get started with the repo today. Repo:

LlamaIndex 🦙

76,371 次观看 • 2 年前

Friends, this Sunday, right before the commencement of Lent, the Church is giving us something of great moment to reflect on—namely, the centrality of freedom and choice for the good at the center of the spiritual life. As Thomas More puts it in “A Man for All Seasons,” “God made animals for innocence and plants for their simplicity. But Man He made to serve Him wittily, in the tangle of his mind.” God wants us to give him glory in a particular way: through our intellect and will—our search for truth and our love for him.

Friends, this Sunday, right before the commencement of Lent, the Church is giving us something of great moment to reflect on—namely, the centrality of freedom and choice for the good at the center of the spiritual life. As Thomas More puts it in “A Man for All Seasons,” “God made animals for innocence and plants for their simplicity. But Man He made to serve Him wittily, in the tangle of his mind.” God wants us to give him glory in a particular way: through our intellect and will—our search for truth and our love for him.

Bishop Robert Barron

44,475 次观看 • 7 个月前

.SnowflakeDB is thrilled to announce #SnowflakeArctic: A state-of-the-art large language model uniquely designed to be the most open, enterprise-grade LLM on the market. This is a big step forward for open source LLMs. And it’s a big moment for Snowflake in our #AI journey as we continue to build best-in-class enterprise-grade products for our customers. The era of enterprise AI is here. 🚀

.SnowflakeDB is thrilled to announce #SnowflakeArctic: A state-of-the-art large language model uniquely designed to be the most open, enterprise-grade LLM on the market. This is a big step forward for open source LLMs. And it’s a big moment for Snowflake in our #AI journey as we continue to build best-in-class enterprise-grade products for our customers. The era of enterprise AI is here. 🚀

sridhar

243,833 次观看 • 2 年前

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.

Jing Yu Koh

124,894 次观看 • 2 年前

We’re thrilled to announce that 2 of our projects are nominated for E-Commerce Site of the Year on awwwards. and that our team is nominated for Studio of the Year for the second time in a row. Take a look at our reel to get a glimpse of the nominated projects & last year’s work!

We’re thrilled to announce that 2 of our projects are nominated for E-Commerce Site of the Year on awwwards. and that our team is nominated for Studio of the Year for the second time in a row. Take a look at our reel to get a glimpse of the nominated projects & last year’s work!

Antinomy Studio

18,451 次观看 • 3 年前

With the UK on course to become the largest contributor of e-waste in the world, we’re highlighting how consumers can tackle it this #EarthDay 🌍 From our O2 Recycle scheme, to our Community Calling initiative, in partnership with Hubbub, we’re supporting the #sustainable disposable of unwanted devices – to avoid them ending up in landfill ♻️ Find out more about our new Virgin TV educational content for #EarthDay2024: Take a look at our commitment to operating in a more sustainable way with our Better Connections Plan:

With the UK on course to become the largest contributor of e-waste in the world, we’re highlighting how consumers can tackle it this #EarthDay 🌍 From our O2 Recycle scheme, to our Community Calling initiative, in partnership with Hubbub, we’re supporting the #sustainable disposable of unwanted devices – to avoid them ending up in landfill ♻️ Find out more about our new Virgin TV educational content for #EarthDay2024: Take a look at our commitment to operating in a more sustainable way with our Better Connections Plan:

Virgin Media O2 News

15,056 次观看 • 2 年前

Insider trading is a feature of Polymarket. A Google employee making $1M+ on search trends is simply efficient price discovery. Platform does not ask if your advantage is fair. It only asks if you are right But this logic leads to a dark place: Assassination Markets. If a market allows bets on a death, it creates a direct financial incentive to... resolve the market When millions are on the line, the platform stops being a passive mirror of the future and becomes an engine for shaping it Are Prediction Markets the most honest and cynical technology of our time?

Insider trading is a feature of Polymarket. A Google employee making $1M+ on search trends is simply efficient price discovery. Platform does not ask if your advantage is fair. It only asks if you are right But this logic leads to a dark place: Assassination Markets. If a market allows bets on a death, it creates a direct financial incentive to... resolve the market When millions are on the line, the platform stops being a passive mirror of the future and becomes an engine for shaping it Are Prediction Markets the most honest and cynical technology of our time?

Lirratø

212,694 次观看 • 9 个月前