Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

🔍 Introducing BackSearch. LLMs are increasingly asked to predict the future, but a good backtest requires a snapshot of the internet at a point in time. BackSearch allows LLMs to search the web as it was on a particular date. It’s great for: 🔮 Forecasting and prediction markets. 📈...

419,098 görüntüleme • 1 ay önce •via X (Twitter)

35 Yorum

will brown profil fotoğrafı
will brown1 ay önce

oh hell yeah this is gonna print

General Reasoning profil fotoğrafı
General Reasoning1 ay önce

Details on how to try BackSearch out here:

Daniel Lougen profil fotoğrafı
Daniel Lougen1 ay önce

Wayback machine for AI, fuck thats smart. I wonder what kind of training data we might get out of this

Sam Z Liu profil fotoğrafı
Sam Z Liu1 ay önce

but upgraded?

Cruncher Jean profil fotoğrafı
Cruncher Jean1 ay önce

You will still inject lookahead bias from the model if it's not completely retrain from scratch. Good for basic RL env but bad for rigorous backtesting. Check ChronosLLM:

Sriraam profil fotoğrafı
Sriraam1 ay önce

We were just talking about this yesterday lol 🔥🔥

Josh Harris profil fotoğrafı
Josh Harris1 ay önce

This is exactly what we need when we were building our frontier financial judgement benchmark - will try to integrate for v2 :)

Vivek Iyer profil fotoğrafı
Vivek Iyer1 ay önce

best data product released in a minute damn

jakedineenasu profil fotoğrafı
jakedineenasu1 ay önce

This is what we target in our paper "Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters", although we look at frozen slices of Reddit as a proxy for general markets against replayed Polymarket events.

Muratcan Koylan profil fotoğrafı
Muratcan Koylan1 ay önce

The timing is perfect; I really needed this.

Albert Catalán Tatjer profil fotoğrafı
Albert Catalán Tatjer1 ay önce

I was just looking for something like this

M profil fotoğrafı
M1 ay önce

watch out ppl are gonna do so much prime-rl on this

λux profil fotoğrafı
λux1 ay önce

this is super cool!! 🙌🏻

Anthony Tamasi profil fotoğrafı
Anthony Tamasi1 ay önce

This is a great idea, but models are pretrained on info up to some cutoff date. Wouldn’t there still be data leakage from future events even without tool use?

prady profil fotoğrafı
prady1 ay önce

Woah!

Alex Peng profil fotoğrafı
Alex Peng1 ay önce

This is so cool

Robin Salimans profil fotoğrafı
Robin Salimans1 ay önce

this looks super useful

Anirudh Ravichandran profil fotoğrafı
Anirudh Ravichandran1 ay önce

excellent, I can totally see the hill climbing with reproducible reward on search environments going brrr @willcb

Mani 🪽 profil fotoğrafı
Mani 🪽1 ay önce

this is super cool !!!!

Vantix AI Agency profil fotoğrafı
Vantix AI Agency1 ay önce

Searching the web from any past date changes how models get tested

Shreyas Pimpalgaonkar profil fotoğrafı
Shreyas Pimpalgaonkar1 ay önce

Wow, this is actually so useful! Congratulations

Nayeem profil fotoğrafı
Nayeem1 ay önce

Incredible

Thomas Mattimore 🇺🇸 profil fotoğrafı
Thomas Mattimore 🇺🇸1 ay önce

Will be a cool way to track various predictions made over time

Restitutor profil fotoğrafı
Restitutor1 ay önce

Awesome, I can think of other uses for this

Oscar Hong profil fotoğrafı
Oscar Hong1 ay önce

this is genius. always wondered why there wasn’t a benchmark for predictions. does this only work for events that happened after a model’s training date?

Matan Halevy profil fotoğrafı
Matan Halevy1 ay önce

unreal, we’re working on a benchmark this would be useful for. Excited to give it a try

Luigi Pagani profil fotoğrafı
Luigi Pagani1 ay önce

Wow, so cool!

Jia Ming (إحسان) profil fotoğrafı
Jia Ming (إحسان)1 ay önce

Google should be doing this

callum profil fotoğrafı
callum1 ay önce

super cool! and very useful

आशीष खरे profil fotoğrafı
आशीष खरे1 ay önce

This is new. I like the general idea. Great one!

Yusuf profil fotoğrafı
Yusuf1 ay önce

Need this

Ha Hoang profil fotoğrafı
Ha Hoang1 ay önce

is this way back machine but for AI agent?

ethan profil fotoğrafı
ethan1 ay önce

great idea

2cents profil fotoğrafı
2cents1 ay önce

@teortaxesTex This is very useful utility

BowTiedStingray profil fotoğrafı
BowTiedStingray1 ay önce

Beautiful - thank you for this

Benzer Videolar

🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. 🤔What's this all about? Why this benchmark? > Back in Nov 2023, when we released MMMU ( a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin ( which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀 > Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent ( showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages. 🌉 Bridging the Gap with VisualWebBench > To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡. 😮 Surprising Findings > 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro. > 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs. > 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent. > 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements. 💡Fun Fact > Claude Sonnet is better than Opus on our benchmark :) 🎓 Conclusion > VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web. Kudos to the student leads Junpeng Liu Yifan Song and the team Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li! 👏 Check out more details in the Junpeng's thread👇

Xiang Yue

56,696 görüntüleme • 2 yıl önce