1/N Introducing🌈PRISM-Physics, a process-level, rule-based benchmark for complex physics reasoning. Each solution is modeled as a Directed Acyclic Graph of formulas, capturing causal relations between steps. A rule-based symbolic equivalence checker ensures consistent evaluation

9/N Website: Paper: Code: Dataset:

N/N Huge thanks to my collaborators and advisors: @AquaHorseM @ShiJingzhe41415 @ShirleyYXWu @jiaqihan99 @YIJIA_XIAO_ @SiYuanChen @XiaoLuo_12 @lschmidt3 @james_y_zou 🫶

4/N Checking physics formula equivalence is challenging due to full-equation matching, constant substitutions and unit conversion. Prior work sidesteps these issues by checking only final expression or using LLM-as-judge, missing process-level reasoning and causes hallucination.

2/N Why DAG? Strict matching is too harsh; prefix credit too permissive. In our DAGs, edges encode prerequisite relations, and credit propagates only along causal chains, from matched formulas to their ancestors, so every rewarded step genuinely contributes to final result.

5/N To address these challenges, we design a two-stage rule-based equivalence checker: [Stage1] Normalize variables, constants, and units into a consistent form. [Stage2] Verify equivalence by solving both equations under random assignments and comparing their solution sets.

6/N Scoring Pipeline Given a student solution and a reference DAG, PRISM-DAG evaluates in 3 steps: 1. Extract and normalize formulas 2. Match them to reference DAG using rule-based Equivalence checker 3. Score via Ancestor Closure Policy, propagating credit along dependencies

3/N Ancestor Closure Scoring We define the ancestor closure of matched formulas to include both the formulas themselves and all their ancestors in the DAG(every prerequisite along the causal chain). The score is the fraction of reference formulas contained in this closure set.

Great work! Thanks Wanjia for leading the project!

Thank you Yijia🫶

7/N Main result Performance declines with difficulty (lower accuracy, longer response time). Final-answer acc drops below 10% on hard problems, revealing sparse reward signals, while step-level scoring better captures process competence and provides dense feedback for training.

8/N We further present additional results, including physics domain category analysis, modality and reasoning-level comparisons, error analysis, and evaluation framework analysis. Please refer to our paper for more details.

What percentage of accuracy are you encountering during the problem decomposition? Would that be the area to improve efficiency on difficult tasks?