Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

🧵🚀 New WebAgent Benchmark Alert! 🚀 There is hope for human workers! We released WorkArena++, a new challenging benchmark for WebAgents. Our best agent achieves 0% accuracy on this benchmark, while human evaluators still obtain 94%! 🔗

alex_lacoste_'s profile picture

Alexandre Lacoste

1,137 subscribers

11,282 Aufrufe • vor 2 Jahren •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

new collab from Paradigm and OpenAI: evmbench is a benchmark and agent harness for exploiting smart contract bugs a few months ago, the best models found <20% of critical, fund-draining @Code4rena bugs in our benchmark. today they find > 70%
0:15

new collab from Paradigm and OpenAI: evmbench is a benchmark and agent harness for exploiting smart contract bugs a few months ago, the best models found <20% of critical, fund-draining @Code4rena bugs in our benchmark. today they find > 70%

Alpin Yukseloglu

226,136 Aufrufe • vor 7 Monaten

We are finally releasing a new benchmark!! #RSS2023 Main highlights are: * For real-world * Reproducible (3D printing) * 9 furniture assembly long-horizon tasks * 200+ hrs of demonstrations This apparently is my 3rd furniture-related benchmark.🥹
0:14

We are finally releasing a new benchmark!! #RSS2023 Main highlights are: * For real-world * Reproducible (3D printing) * 9 furniture assembly long-horizon tasks * 200+ hrs of demonstrations This apparently is my 3rd furniture-related benchmark.🥹

Joseph Lim

27,041 Aufrufe • vor 3 Jahren

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.
0:18

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.

AK

19,684 Aufrufe • vor 2 Jahren

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang
0:15

Introduce EAGLE, a new method for fast LLM decoding based on compression: - 3x🚀than vanilla - 2x🚀 than Lookahead (on its benchmark) - 1.6x🚀 than Medusa (on its benchmark) - provably maintains text distribution - trainable (in 1~2 days) and testable on RTX 3090s Playground: Blog: Code: ⚒️First Principle: Compression! Yi Ma We find that the sequence of second-top-layer features is compressible, making the prediction of subsequent feature vectors from previous ones easy by a small model. 🙏Acknowledge: This project is greatly inspired by the Medusa team (Tianle Cai @yli3521 Zhengyang Geng Hongwu Peng Tri Dao), the Lookahead team (Hao Zhang LMSYS Org), and others. Joint work with Yuhui Li and Chao Zhang

Hongyang Zhang

118,857 Aufrufe • vor 2 Jahren

The bar for African language TTS has just been raised 🇬🇭 Listen to the quality of these new Asante (male AND female) TTS voices. This is the natural, authentic sound being worked towards. Genuinely believe this sets a new benchmark. This is a huge opportunity for integration. Let's give our apps the voice they deserve. Link in cs 👇
0:10

The bar for African language TTS has just been raised 🇬🇭 Listen to the quality of these new Asante (male AND female) TTS voices. This is the natural, authentic sound being worked towards. Genuinely believe this sets a new benchmark. This is a huge opportunity for integration. Let's give our apps the voice they deserve. Link in cs 👇

Prince Larbi

30,644 Aufrufe • vor 10 Monaten

We are hiring FT SW/ML and Mechanical roles at Human Computer Lab Robots aren't just for laundry and dishes in homes. LeLamp has a unique constraint today: it doesn't move through space or manipulate objects. That limitation pushes us to think deeply about how LeLamp can understand human intent and translate it into trajectories that make it feel alive. This makes us singularly focused on defining a new benchmark for interacting with technology — one that feels joyful and makes us more aware of our relationship to it.
0:12

We are hiring FT SW/ML and Mechanical roles at Human Computer Lab Robots aren't just for laundry and dishes in homes. LeLamp has a unique constraint today: it doesn't move through space or manipulate objects. That limitation pushes us to think deeply about how LeLamp can understand human intent and translate it into trajectories that make it feel alive. This makes us singularly focused on defining a new benchmark for interacting with technology — one that feels joyful and makes us more aware of our relationship to it.

Shahvir Sarkary

31,550 Aufrufe • vor 5 Monaten

Convolutional neural nets going to spacetime 🚀 Our new ICML24 paper shows how to build Lorentz-equivariant CNNs/MPNNs for multivector fields on Minkowski spaces. This is useful for particle physics or Navier Stokes / electrodynamics simulations. 🧵1/N
0:13

Convolutional neural nets going to spacetime 🚀 Our new ICML24 paper shows how to build Lorentz-equivariant CNNs/MPNNs for multivector fields on Minkowski spaces. This is useful for particle physics or Navier Stokes / electrodynamics simulations. 🧵1/N

Maurice Weiler

266,639 Aufrufe • vor 2 Jahren

Announcing Bespoke-MiniChart-7B, a new SOTA in chart understanding for models of comparable size on seven benchmarks, on par with Gemini-1.5-Pro and Claude-3.5! 🚀 Beyond its real-world applications, chart understanding is a good challenging problem for VLMs, since it requires both mathematical as well as visual reasoning. 1/n🧵
0:14

Announcing Bespoke-MiniChart-7B, a new SOTA in chart understanding for models of comparable size on seven benchmarks, on par with Gemini-1.5-Pro and Claude-3.5! 🚀 Beyond its real-world applications, chart understanding is a good challenging problem for VLMs, since it requires both mathematical as well as visual reasoning. 1/n🧵

Bespoke Labs

19,871 Aufrufe • vor 1 Jahr

📢 AITECH Data Center is Now Live! Solidus Ai Tech’s HPC Data Center is now live — the first ever built by a Web3 company, setting a new benchmark for decentralized AI infrastructure. 8000 sqft | 20MW capacity | 624 TFLOPS compute | RADOX cooling | <1ms latency | ISO 27001 compliant Next up is the launch of the Compute Marketplace, unlocking access to our available compute — all powered by the $AITECH token. A massive leap forward. 🔗 Learn more:
0:17

📢 AITECH Data Center is Now Live! Solidus Ai Tech’s HPC Data Center is now live — the first ever built by a Web3 company, setting a new benchmark for decentralized AI infrastructure. 8000 sqft | 20MW capacity | 624 TFLOPS compute | RADOX cooling | <1ms latency | ISO 27001 compliant Next up is the launch of the Compute Marketplace, unlocking access to our available compute — all powered by the $AITECH token. A massive leap forward. 🔗 Learn more:

AITECH CLOUD NETWORK

170,573 Aufrufe • vor 1 Jahr

Hey #Catchers👽 Today we have prepared for you a small sample of the direction in which our development has moved. We're still working on the animations, but our Combat #AR is almost complete🚀 We hope you will appreciate #SpaceCatch progress 🔥 #PoGoKiller #Crypto #P2EGame
0:07

Hey #Catchers👽 Today we have prepared for you a small sample of the direction in which our development has moved. We're still working on the animations, but our Combat #AR is almost complete🚀 We hope you will appreciate #SpaceCatch progress 🔥 #PoGoKiller #Crypto #P2EGame

SpaceCatch

224,603 Aufrufe • vor 3 Jahren

The AI hunt for alien life has just begun. Welcome to ThousandsWorlds, a wild new dataset from researchers at Oxford/Cambridge++, for detecting faint signatures in the atmospheres of potentially habitable exoplanets. This is the first step towards finding life beyond earth. The plan is basically: 1) scan the galaxy for as many potentially habitable planets as possible 2) detect the gases in their atmospheres with powerful telescopes like JWST 3) infer from these gases whether life is present or not. ThousandWorlds is a benchmark for emulating these exoplanet climates: 1760 simulations across 5 GCMs, 8 planet parameters, and atmospheric variables on a 32 x 64 x 10 latitude-longitude-pressure grid. It includes three nested benchmark subsets, two evaluation protocols, and eight released baseline methods. incredible work from Miles Cranmer and many more 👽👽👽
0:11

The AI hunt for alien life has just begun. Welcome to ThousandsWorlds, a wild new dataset from researchers at Oxford/Cambridge++, for detecting faint signatures in the atmospheres of potentially habitable exoplanets. This is the first step towards finding life beyond earth. The plan is basically: 1) scan the galaxy for as many potentially habitable planets as possible 2) detect the gases in their atmospheres with powerful telescopes like JWST 3) infer from these gases whether life is present or not. ThousandWorlds is a benchmark for emulating these exoplanet climates: 1760 simulations across 5 GCMs, 8 planet parameters, and atmospheric variables on a 32 x 64 x 10 latitude-longitude-pressure grid. It includes three nested benchmark subsets, two evaluation protocols, and eight released baseline methods. incredible work from Miles Cranmer and many more 👽👽👽

Georgia Channing

19,694 Aufrufe • vor 3 Monaten

NEW FEATURES REVEAL ✨🚀 Tune into #CampStrava2024 LIVE on May 16 at 9:45am PT on our YouTube channel for a first look at the new features and partnerships coming to Strava this year. Set a reminder now – we think you'll like what we've cooked up. 🍳
0:17

NEW FEATURES REVEAL ✨🚀 Tune into #CampStrava2024 LIVE on May 16 at 9:45am PT on our YouTube channel for a first look at the new features and partnerships coming to Strava this year. Set a reminder now – we think you'll like what we've cooked up. 🍳

Strava

13,640 Aufrufe • vor 2 Jahren

You can swap your old $MEMEX tokens for the new $MMX tokens via 🔗 This transition marks the beginning of a brand new era for the MEMEXAI ecosystem 🚀 $MMX is now listed and tradable on IndoEx. The future is taking shape 👀🔥 #MMX $MMX #MEMEX #MEMEXAI IndoEx Electra Protocol Pollarea
0:13

You can swap your old $MEMEX tokens for the new $MMX tokens via 🔗 This transition marks the beginning of a brand new era for the MEMEXAI ecosystem 🚀 $MMX is now listed and tradable on IndoEx. The future is taking shape 👀🔥 #MMX $MMX #MEMEX #MEMEXAI IndoEx Electra Protocol Pollarea

Noble Eagle

1,196,680 Aufrufe • vor 8 Monaten

🫧🏠 One of the houses in my neighborhood has a bubble machine that they turn on every once in a while, pushing up the dial on joy for every human and pet in the vicinity and anyhow, this is a new life goal
0:13

🫧🏠 One of the houses in my neighborhood has a bubble machine that they turn on every once in a while, pushing up the dial on joy for every human and pet in the vicinity and anyhow, this is a new life goal

👩‍💻 Paige Bailey

11,290 Aufrufe • vor 3 Monaten

Dear Survivors, Escape To Hope 🕊️ We're off-planet. We saved everyone we could. As we leave the ruins of Archiesland behind us, voyaging towards this blue marble we hope to call our new home, we know we can carry hope in our hearts for one reason. We will live on. Why? Because we're carrying our very future on this vessel - thousands of Archie eggs! There is light that pierces the darkness, illuminating the way forward. We're on our way to Earth 🌍⤵️
0:10

Dear Survivors, Escape To Hope 🕊️ We're off-planet. We saved everyone we could. As we leave the ruins of Archiesland behind us, voyaging towards this blue marble we hope to call our new home, we know we can carry hope in our hearts for one reason. We will live on. Why? Because we're carrying our very future on this vessel - thousands of Archie eggs! There is light that pierces the darkness, illuminating the way forward. We're on our way to Earth 🌍⤵️

Archies NFT 🟠⛺️

10,710 Aufrufe • vor 2 Jahren

Constructing interactive simulated worlds has been a challenging problem, requiring considerable manual effort for asset creation and articulation, and composing assets to form full scenes. In our new work - DRAWER, we made the process of creating scenes in simulation as simple as taking a video of the scene and out comes a high-quality, fully interactive environment in simulation. No human simulation designer involved! A 🧵(1/7)
0:15

Constructing interactive simulated worlds has been a challenging problem, requiring considerable manual effort for asset creation and articulation, and composing assets to form full scenes. In our new work - DRAWER, we made the process of creating scenes in simulation as simple as taking a video of the scene and out comes a high-quality, fully interactive environment in simulation. No human simulation designer involved! A 🧵(1/7)

Abhishek Gupta

12,086 Aufrufe • vor 1 Jahr

LongWriter Unleashing 10,000+ Word Generation from Long Context LLMs discuss: Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability.
0:12

LongWriter Unleashing 10,000+ Word Generation from Long Context LLMs discuss: Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability.

AK

50,995 Aufrufe • vor 2 Jahren

The countdown is on..... ⏰ T-minus 2 days until we find out who will be OUR NEW CO-HOST!!!!!!!! *ahem* sorry, we lost our cool there for a second... This Monday's show... 1 pm on CTV... don't miss it 👀
0:12

The countdown is on..... ⏰ T-minus 2 days until we find out who will be OUR NEW CO-HOST!!!!!!!! *ahem* sorry, we lost our cool there for a second... This Monday's show... 1 pm on CTV... don't miss it 👀

The Social

18,156 Aufrufe • vor 3 Jahren