Evaluation

Measured, not claimed

500 seeded synthetic learners, up to 6 questions each. Seed 2026. Regenerate with pnpm eval.

Diagnosis

GLITCH

right 500wrong 0missed 0q 2

Random next question

right 442wrong 1missed 57q 4

Wrong answer lookup

right 420wrong 0missed 80q 4

Answers only, no steps

right 498wrong 2missed 0q 2

Right means a boss for the learner’s real misconception, or no boss when there was none. Wrong includes giving a boss to a learner with no misconception. q is the median number of questions before a boss appeared.

No boss when no single rule fits

Mixed rules, one per problem321 given a boss179 of 500
Random answers21 given a boss479 of 500
Careless, slips half the timenever given a boss500 of 500

Reliability

Math truth failures4995 problems checked0
Correct rule disagreementsrule language vs truth engine0
Invalid programs accepted15 malformed programs tried0
False counterexamples3240 counterexamples checked0
Stage order violations accepted60000 random completion attempts, checked independently0

Latency

Choosing the next question, P500.025 ms
Choosing the next question, P95budget 150 ms0.033 ms

Live model

Modellow reasoning effortgpt-5.5
Runs that found a verified rule10 of 10
Matched the true rule on unseen problems90 problems per run10 of 10
Proposed programs rejected0 of 19
Explanations judged as labelled8 of 9
Finding a rule, P50 / P95goal 2.5 s5.5 / 9.7 s
Reading an explanation, P50 / P952.7 / 5.7 s

Known limits

  • The synthetic learners follow the same four rules the model knows, so these numbers show the machinery works, not how accurate it is with real children.
  • Free Ten and correct regrouping give the same answer when no regrouping is needed. GLITCH will not wake a boss until it has seen two problems where they disagree.
  • The model is slower than intended: finding a new rule takes several seconds, not the 2.5 second goal.
  • A learner who switches between rules problem by problem still gets a boss most of the time: the gate checks that a rule fits two telling answers, not that it explains the rest.
  • The model eval is small: ten rule-finding runs over two rules and nine hand-labelled explanations.