Hongjian Zhou's banner

Hongjian Zhou

@itsEmZee_ • 4,341 subscribers

PhD @UniOfOxford, Clarendon Scholar. I work on AI scientists for medicine and biomedical research. Opinions are my own.

Shorts

Can LLMs keep correct medical judgment when the context is misleading? LLMs ace medical exams. But real-world use cases rarely come with clean context the way exams do. So we investigated whether LLMs preserve correct medical judgment under misleading context. Our findings: > Across 11 frontier model configs, accuracy dropped from 71% to 38%. > A 14-member clinical panel from 7 countries identified serious potential harm in 38.2% of reviewed cases. 1/8

Can LLMs keep correct medical judgment when the context is misleading? LLMs ace medical exams. But real-world use cases rarely come with clean context the way exams do. So we investigated whether LLMs preserve correct medical judgment under misleading context. Our findings: > Across 11 frontier model configs, accuracy dropped from 71% to 38%. > A 14-member clinical panel from 7 countries identified serious potential harm in 38.2% of reviewed cases. 1/8

52,730 views