Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

🧵🚀 New WebAgent Benchmark Alert! 🚀 There is hope for human workers! We released WorkArena++, a new challenging benchmark for WebAgents. Our best agent achieves 0% accuracy on this benchmark, while human evaluators still obtain 94%! 🔗

Alexandre Lacoste

1,139 subscribers

11,271 views • 1 year ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

new collab from Paradigm and OpenAI: evmbench is a benchmark and agent harness for exploiting smart contract bugs a few months ago, the best models found <20% of critical, fund-draining @Code4rena bugs in our benchmark. today they find > 70%

new collab from Paradigm and OpenAI: evmbench is a benchmark and agent harness for exploiting smart contract bugs a few months ago, the best models found <20% of critical, fund-draining @Code4rena bugs in our benchmark. today they find > 70%

Alpin Yukseloglu @ ICML

224,666 views • 5 months ago

We are finally releasing a new benchmark!! #RSS2023 Main highlights are: * For real-world * Reproducible (3D printing) * 9 furniture assembly long-horizon tasks * 200+ hrs of demonstrations This apparently is my 3rd furniture-related benchmark.🥹

We are finally releasing a new benchmark!! #RSS2023 Main highlights are: * For real-world * Reproducible (3D printing) * 9 furniture assembly long-horizon tasks * 200+ hrs of demonstrations This apparently is my 3rd furniture-related benchmark.🥹

Joseph Lim

26,947 views • 3 years ago

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.

AK

19,684 views • 1 year ago

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang

Hongyang Zhang

118,857 views • 2 years ago

🚀 Top4Fans V2 is now live ! This is just the beginning, with more updates and new features on the way for your best experience as we enter a new era for Top4Fans! 💥

🚀 Top4Fans V2 is now live ! This is just the beginning, with more updates and new features on the way for your best experience as we enter a new era for Top4Fans! 💥

TOP4FANS

17,414 views • 1 year ago

The bar for African language TTS has just been raised 🇬🇭 Listen to the quality of these new Asante (male AND female) TTS voices. This is the natural, authentic sound being worked towards. Genuinely believe this sets a new benchmark. This is a huge opportunity for integration. Let's give our apps the voice they deserve. Link in cs 👇

The bar for African language TTS has just been raised 🇬🇭 Listen to the quality of these new Asante (male AND female) TTS voices. This is the natural, authentic sound being worked towards. Genuinely believe this sets a new benchmark. This is a huge opportunity for integration. Let's give our apps the voice they deserve. Link in cs 👇

larbi

30,630 views • 9 months ago

We are hiring FT SW/ML and Mechanical roles at Human Computer Lab Robots aren't just for laundry and dishes in homes. LeLamp has a unique constraint today: it doesn't move through space or manipulate objects. That limitation pushes us to think deeply about how LeLamp can understand human intent and translate it into trajectories that make it feel alive. This makes us singularly focused on defining a new benchmark for interacting with technology — one that feels joyful and makes us more aware of our relationship to it.

We are hiring FT SW/ML and Mechanical roles at Human Computer Lab Robots aren't just for laundry and dishes in homes. LeLamp has a unique constraint today: it doesn't move through space or manipulate objects. That limitation pushes us to think deeply about how LeLamp can understand human intent and translate it into trajectories that make it feel alive. This makes us singularly focused on defining a new benchmark for interacting with technology — one that feels joyful and makes us more aware of our relationship to it.

Shahvir Sarkary

31,550 views • 3 months ago

Convolutional neural nets going to spacetime 🚀 Our new ICML24 paper shows how to build Lorentz-equivariant CNNs/MPNNs for multivector fields on Minkowski spaces. This is useful for particle physics or Navier Stokes / electrodynamics simulations. 🧵1/N

Convolutional neural nets going to spacetime 🚀 Our new ICML24 paper shows how to build Lorentz-equivariant CNNs/MPNNs for multivector fields on Minkowski spaces. This is useful for particle physics or Navier Stokes / electrodynamics simulations. 🧵1/N

Maurice Weiler

266,567 views • 2 years ago

Announcing Bespoke-MiniChart-7B, a new SOTA in chart understanding for models of comparable size on seven benchmarks, on par with Gemini-1.5-Pro and Claude-3.5! 🚀 Beyond its real-world applications, chart understanding is a good challenging problem for VLMs, since it requires both mathematical as well as visual reasoning. 1/n🧵

Announcing Bespoke-MiniChart-7B, a new SOTA in chart understanding for models of comparable size on seven benchmarks, on par with Gemini-1.5-Pro and Claude-3.5! 🚀 Beyond its real-world applications, chart understanding is a good challenging problem for VLMs, since it requires both mathematical as well as visual reasoning. 1/n🧵

Bespoke Labs

19,871 views • 1 year ago

📢 AITECH Data Center is Now Live! Solidus Ai Tech’s HPC Data Center is now live — the first ever built by a Web3 company, setting a new benchmark for decentralized AI infrastructure. 8000 sqft | 20MW capacity | 624 TFLOPS compute | RADOX cooling | <1ms latency | ISO 27001 compliant Next up is the launch of the Compute Marketplace, unlocking access to our available compute — all powered by the $AITECH token. A massive leap forward. 🔗 Learn more:

📢 AITECH Data Center is Now Live! Solidus Ai Tech’s HPC Data Center is now live — the first ever built by a Web3 company, setting a new benchmark for decentralized AI infrastructure. 8000 sqft | 20MW capacity | 624 TFLOPS compute | RADOX cooling | <1ms latency | ISO 27001 compliant Next up is the launch of the Compute Marketplace, unlocking access to our available compute — all powered by the $AITECH token. A massive leap forward. 🔗 Learn more:

AITECH CLOUD NETWORK

170,573 views • 1 year ago

Hey #Catchers👽 Today we have prepared for you a small sample of the direction in which our development has moved. We're still working on the animations, but our Combat #AR is almost complete🚀 We hope you will appreciate #SpaceCatch progress 🔥 #PoGoKiller #Crypto #P2EGame

Hey #Catchers👽 Today we have prepared for you a small sample of the direction in which our development has moved. We're still working on the animations, but our Combat #AR is almost complete🚀 We hope you will appreciate #SpaceCatch progress 🔥 #PoGoKiller #Crypto #P2EGame

SpaceCatch

224,603 views • 2 years ago

The AI hunt for alien life has just begun. Welcome to ThousandsWorlds, a wild new dataset from researchers at Oxford/Cambridge++, for detecting faint signatures in the atmospheres of potentially habitable exoplanets. This is the first step towards finding life beyond earth. The plan is basically: 1) scan the galaxy for as many potentially habitable planets as possible 2) detect the gases in their atmospheres with powerful telescopes like JWST 3) infer from these gases whether life is present or not. ThousandWorlds is a benchmark for emulating these exoplanet climates: 1760 simulations across 5 GCMs, 8 planet parameters, and atmospheric variables on a 32 x 64 x 10 latitude-longitude-pressure grid. It includes three nested benchmark subsets, two evaluation protocols, and eight released baseline methods. incredible work from Miles Cranmer and many more 👽👽👽

The AI hunt for alien life has just begun. Welcome to ThousandsWorlds, a wild new dataset from researchers at Oxford/Cambridge++, for detecting faint signatures in the atmospheres of potentially habitable exoplanets. This is the first step towards finding life beyond earth. The plan is basically: 1) scan the galaxy for as many potentially habitable planets as possible 2) detect the gases in their atmospheres with powerful telescopes like JWST 3) infer from these gases whether life is present or not. ThousandWorlds is a benchmark for emulating these exoplanet climates: 1760 simulations across 5 GCMs, 8 planet parameters, and atmospheric variables on a 32 x 64 x 10 latitude-longitude-pressure grid. It includes three nested benchmark subsets, two evaluation protocols, and eight released baseline methods. incredible work from Miles Cranmer and many more 👽👽👽

Georgia Channing

18,885 views • 1 month ago

🚀 Exciting times for the $BRN community as we continue to grow, evolve, and push the boundaries! 🌟 Catch $BRN on: Gate.io @MEXC_Official BitMart XT Exchange 🚀 And we're not stopping there! This month, we're thrilled to announce a new exchange listing! 📈 Are you ready for more? #GameFi #WEB3 #cryptocurrency

🚀 Exciting times for the $BRN community as we continue to grow, evolve, and push the boundaries! 🌟 Catch $BRN on: Gate.io @MEXC_Official BitMart XT Exchange 🚀 And we're not stopping there! This month, we're thrilled to announce a new exchange listing! 📈 Are you ready for more? #GameFi #WEB3 #cryptocurrency

BRN Metaverse

83,618 views • 1 year ago

NEW FEATURES REVEAL ✨🚀 Tune into #CampStrava2024 LIVE on May 16 at 9:45am PT on our YouTube channel for a first look at the new features and partnerships coming to Strava this year. Set a reminder now – we think you'll like what we've cooked up. 🍳

NEW FEATURES REVEAL ✨🚀 Tune into #CampStrava2024 LIVE on May 16 at 9:45am PT on our YouTube channel for a first look at the new features and partnerships coming to Strava this year. Set a reminder now – we think you'll like what we've cooked up. 🍳

Strava

13,603 views • 2 years ago

You can swap your old $MEMEX tokens for the new $MMX tokens via 🔗 This transition marks the beginning of a brand new era for the MEMEXAI ecosystem 🚀 $MMX is now listed and tradable on IndoEx. The future is taking shape 👀🔥 #MMX $MMX #MEMEX #MEMEXAI IndoEx Electra Protocol Pollarea

Noble Eagle

1,196,680 views • 6 months ago

🫧🏠 One of the houses in my neighborhood has a bubble machine that they turn on every once in a while, pushing up the dial on joy for every human and pet in the vicinity and anyhow, this is a new life goal

🫧🏠 One of the houses in my neighborhood has a bubble machine that they turn on every once in a while, pushing up the dial on joy for every human and pet in the vicinity and anyhow, this is a new life goal

👩‍💻 Paige Bailey

11,290 views • 1 month ago

🚀 Pro Just Got More Powerful Today we're releasing ‘Expert Mode' for WeatherWise Pro subscribers Turn on multiple modes, like Radar and Satellite, at the same time - Synchronized playback across all the activated modes - Data picker works across multiple modes - All modes automatically update with the latest data There is more... We are also adding: - Smoothing for the Radar Cross Section tool - Triple View for Tablet/Desktop - A new scan alert sound for FastScan - Keyboard shortcut for Reflectivity X - Improved smoothing for Composite imagery - Check out everything new in this thread! 🧵 Upgrade to version 1.4.0: iOS: Android: Available now in the web browser and through our Mac, Windows and Linux App

🚀 Pro Just Got More Powerful Today we're releasing ‘Expert Mode' for WeatherWise Pro subscribers Turn on multiple modes, like Radar and Satellite, at the same time - Synchronized playback across all the activated modes - Data picker works across multiple modes - All modes automatically update with the latest data There is more... We are also adding: - Smoothing for the Radar Cross Section tool - Triple View for Tablet/Desktop - A new scan alert sound for FastScan - Keyboard shortcut for Reflectivity X - Improved smoothing for Composite imagery - Check out everything new in this thread! 🧵 Upgrade to version 1.4.0: iOS: Android: Available now in the web browser and through our Mac, Windows and Linux App

WeatherWise.app

21,902 views • 11 months ago

Dear Survivors, Escape To Hope 🕊️ We're off-planet. We saved everyone we could. As we leave the ruins of Archiesland behind us, voyaging towards this blue marble we hope to call our new home, we know we can carry hope in our hearts for one reason. We will live on. Why? Because we're carrying our very future on this vessel - thousands of Archie eggs! There is light that pierces the darkness, illuminating the way forward. We're on our way to Earth 🌍⤵️

Dear Survivors, Escape To Hope 🕊️ We're off-planet. We saved everyone we could. As we leave the ruins of Archiesland behind us, voyaging towards this blue marble we hope to call our new home, we know we can carry hope in our hearts for one reason. We will live on. Why? Because we're carrying our very future on this vessel - thousands of Archie eggs! There is light that pierces the darkness, illuminating the way forward. We're on our way to Earth 🌍⤵️

Archies NFT 🟠⛺️

10,708 views • 2 years ago

Constructing interactive simulated worlds has been a challenging problem, requiring considerable manual effort for asset creation and articulation, and composing assets to form full scenes. In our new work - DRAWER, we made the process of creating scenes in simulation as simple as taking a video of the scene and out comes a high-quality, fully interactive environment in simulation. No human simulation designer involved! A 🧵(1/7)

Constructing interactive simulated worlds has been a challenging problem, requiring considerable manual effort for asset creation and articulation, and composing assets to form full scenes. In our new work - DRAWER, we made the process of creating scenes in simulation as simple as taking a video of the scene and out comes a high-quality, fully interactive environment in simulation. No human simulation designer involved! A 🧵(1/7)

Abhishek Gupta

12,072 views • 1 year ago

LongWriter Unleashing 10,000+ Word Generation from Long Context LLMs discuss: Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability.

LongWriter Unleashing 10,000+ Word Generation from Long Context LLMs discuss: Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability.

AK

50,995 views • 2 years ago