Databricks Chief AI Scientist Jonathan Frankle (Jonathan Frankle) says the bottleneck to enterprise AI adoption is evals, not models. "The models are good enough at a lot of really complex enterprise tasks and knowledge tasks" "the real challenge is describing what you want AI to do and knowing whether... AI did it"show more

@jefrankle @mercor @BrendanFoody Corporate politics is the biggest bottleneck. The solution to adoption is mayhem.

@jefrankle @mercor @BrendanFoody haha true

Agree. The 'describe what you want' half is mostly a modelling problem. If the business entities, states and rules aren't written down anywhere (an ontology, basically), you can't write evals that mean anything to the business. Do you see clients who focus on modelling business logic rather than dumping more text into the agent?

@thinkshiv @jefrankle @mercor @BrendanFoody Somewhat. More that the execution path is blocked or not discoverable (different apis, mcp, data lakes, tools, systems, clouds).

@jefrankle @mercor @BrendanFoody The models aren't good enough yet. They still hallucinate on enterprise data and fail at multi-step reasoning under pressure. Both model quality and evals need work, not one or the other 🤷‍♂️

@jefrankle @mercor @BrendanFoody Describing what you want is basically writing a spec, and most teams never wrote one for knowledge work because a person quietly filled the gaps. The eval ends up being the first honest job description.

@jefrankle @mercor @BrendanFoody Welcome to me TED talk

@jefrankle @mercor @BrendanFoody Agree. Once models are good enough, the hard part is defining 'correct' for your own tasks and checking it every time. Evals built on your own data are what turn a demo into something a team can trust

@jefrankle @mercor @BrendanFoody モデルの精度だけ上げても、現場の依頼をどう評価するかが曖昧だと導入判断できないんですよね。まず「何をもって完了か」と例外時の扱いを決めて、小さく実データで確かめるのが結局近道だと思います。

full pod

@jefrankle @mercor @BrendanFoody i'd give the 'knowing whether ai did it' half to the people who did the task by hand, not a central ai team. the person who ran the month-end reconciliation for years can spot a wrong agent output in a minute.

@jefrankle @mercor @BrendanFoody @mikeysee 👀

@jefrankle @mercor @BrendanFoody describing what you want is the part nobody budgets time for most teams can't say what a good output looks like until they've seen a bad one lol

@jefrankle @mercor @BrendanFoody Yep common for small biz getting at to do new things

@jefrankle @mercor @BrendanFoody yeah and small biz has it worse because the owner is usually the only one who knows what good looks like do they get clearer after the first bad batch or just more annoyed

@jefrankle @mercor @BrendanFoody He wanted to say "verification" I guess.

@jefrankle @mercor @BrendanFoody So much institutional knowledge is muscle memory for tenured employees. Things they’d never think to put in a prompt. So the ai literally has a partial script to follow and 0 experience. Results won’t follow this.

@jefrankle @mercor @BrendanFoody same thing in outbound. the writing is fine, nobody wrote down who the buyer actually is. describing the target is most of the job

@jefrankle @mercor @BrendanFoody describing what you want is the hard part for people too tbh. half the briefs i've seen handed to a new hire would fail an eval

Describing the work is the hard part. Rethink it from first principles with AI at the center, or you end up with a pilot that never makes it to production. Then put the exceptions and ground truth in place as evals, so you can measure whether it works. This is where forward deployed engineers earn their place. They're curious, and they surface the deep organizational truth that ops teams never have time to write down.

@jefrankle @mercor @BrendanFoody That’s weird take… if you give smartest person goal he will find a way and define all things… What a nonesense these people say

@jefrankle @mercor @BrendanFoody On the "knowing whether AI did it" half: run each case more than once. IBM Research's ALTK-Evolve write-up had a retail agent averaging 78.1%, but 57.5% when a task only counts if all 4 runs succeed. Enterprises live with the second number.

@jefrankle @mercor @BrendanFoody Models are often good enough. What’s scarce is knowing exactly what “good” looks like and being able to measure it. Like having a skilled cook but no one who can taste the food. Judgment is the current bottleneck.