正在加载视频...

视频加载失败

Introducing the Open Axis Benchmark, a living benchmark engine for honest evaluation of robot manipulation models, built together with OpenRoboto. Robotic models evolve faster every month, while most benchmarks stay frozen. Models overfit to fixed task sets, scores stop reflecting real generalization, and that distorted signal misleads and holds...

43,337 次观看 • 1 个月前 •via X (Twitter)

35 条评论

Chris P. Bacon ❪✱,✱❫ 的头像
Chris P. Bacon ❪✱,✱❫1 个月前

@openroboto

Biswa FF 的头像
Biswa FF1 个月前

@openroboto finally a benchmark that actually keeps up with model progress

Md. Mehedi Hasan 的头像
Md. Mehedi Hasan1 个月前

@openroboto That’s the way to do it! 🎯 The Axis to Jumper pipeline is looking strong. Keep riding that alpha wave! 🌊

Zyrella 的头像
Zyrella1 个月前

@openroboto Fresh task sets make robot evaluation much more meaningful

LanrewajuCrypto🦇 的头像
LanrewajuCrypto🦇1 个月前

@openroboto You're doing well

Maxi 的头像
Maxi1 个月前

@openroboto Dope addition tbh.

0xOnderfhm 的头像
0xOnderfhm1 个月前

@openroboto gAxis

anar.eth 的头像
anar.eth1 个月前

@openroboto interesting

Shaon ☠️ eth 🔥 的头像
Shaon ☠️ eth 🔥1 个月前

@openroboto gAxis

william 的头像
william1 个月前

@openroboto time to explore

Danh Tran 🔶 的头像
Danh Tran 🔶1 个月前

@openroboto bullish Axis 🙌🙌

SAN.SOMETHING 的头像
SAN.SOMETHING1 个月前

@openroboto That's huge

Sydney (✱,✱) 的头像
Sydney (✱,✱)1 个月前

@openroboto up up up

papzzzicle(✱,✱) 的头像
papzzzicle(✱,✱)1 个月前

@openroboto Been grinding Axis robotics task and I would probably say that Libero tasks are challenging and good for learning the mastery in controlling the robot arms

1% 的头像
1%1 个月前

@openroboto let’s explore

N𝕒𝕫 🫟 的头像
N𝕒𝕫 🫟1 个月前

@openroboto this is amazing

rez (✱,✱) 的头像
rez (✱,✱)1 个月前

@openroboto old benchmarks jjust train robots to memorize

Orio_Trade | Web3 的头像
Orio_Trade | Web31 个月前

@openroboto Good to know that robotics models evolve faster every month

0xbossj (✱,✱) 的头像
0xbossj (✱,✱)1 个月前

@openroboto this is bullish! LFAxis!

Sraboni💎⚡️ 的头像
Sraboni💎⚡️1 个月前

@openroboto Axis on fire

WAi XeA 的头像
WAi XeA1 个月前

@openroboto a living eval for robot hands just shipped. the scoreboard finally has a home

ZOZA 的头像
ZOZA1 个月前

@openroboto Wow

hoko 的头像
hoko1 个月前

@openroboto gAxis

ryzzu 的头像
ryzzu1 个月前

@openroboto gAxis

Akash 的头像
Akash1 个月前

@openroboto optimize for specific task parameters rather than generalized spatial reasoning

🟠Fycee (✱,✱) | CryptitaPlays🛸 的头像
🟠Fycee (✱,✱) | CryptitaPlays🛸1 个月前

@openroboto LFG!!!

Future Network 的头像
Future Network1 个月前

@openroboto Each task will have a reward pool paid in USD (so the token doesn't get inflated). Whoever performs the tasks better will earn more dollars. We need to make sure the reward pool still offers a profitable ratio so that people stay excited to join.

Kiva 🏴‍☠️ 的头像
Kiva 🏴‍☠️1 个月前

@openroboto impressive work, countless hours fueling progress

Axis Robotics Philippines 的头像
Axis Robotics Philippines1 个月前

@openroboto Sobrang dami nating matututunan dito! Ang lakas talaga ni @openroboto!

Macro Bombastic 的头像
Macro Bombastic1 个月前

@openroboto honestly fresh task sets are the only way to know it actually generalizes

0xkenyaz 的头像
0xkenyaz1 个月前

@openroboto frozen task sets get gamed so fast tho

Nycole 的头像
Nycole1 个月前

@openroboto fresh tasks make benchmarks way more meaningful

Vasu 的头像
Vasu1 个月前

@openroboto axis bigger picture is here...

MD. Khalid 的头像
MD. Khalid1 个月前

@openroboto gAxis

𝑺𝒂𝒓𝒊𝒉𝒂 𝑱𝒂𝒏𝒏𝒂𝒕 的头像
𝑺𝒂𝒓𝒊𝒉𝒂 𝑱𝒂𝒏𝒏𝒂𝒕1 个月前

@openroboto Continuous evaluation could make robotic progress more measurable

相关视频

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

133,952 次观看 • 3 个月前

Small Language Models (SML) are the future of AI. "Small" (SML) instead of "Large" (LLM). These small models are highly specialized models with superhuman abilities on specific tasks. Here are two techniques to build these models: • Spectrum • Model Merging I give you a short introduction in the attached video, but here is a quick summary: Spectrum helps us identify the most relevant layers to solve one specific task. We can ignore everything else and focus on fine-tuning these layers. Using Spectrum, we can fine-tune models in a heartbeat. Model Merging combines multiple models into a unique, much better model than any of the individual input models. You can also combine models specialized in different tasks and get a model with multiple abilities. This is the state of the art of productizing models. It's what Arcee.ai's platform does behind the scenes. Arcee collaborated with me on this post and is sponsoring it. There are three main steps to produce a model for your particular use case: 1. You create a dataset by uploading your data. 2. You train a model. At this step, Arcee uses Spectrum and Model Merging to produce a highly specialized model for your task. 3. You can deploy that model to any environment you want. Three important notes: • Training process is 2x faster and 2x cheaper than regular fine-tuning. • Resultant models are smaller and have higher accuracy. • They create these specialized models from open-source models. Check this site so you can fully appreciate how this works: If you want to fine-tune an open-source model, consider Arcee's platform. This is the state of the art.

Santiago

164,162 次观看 • 2 年前