📐 Reliability Diagram
When the brain says 70%, does it really win 70%? · the gold-standard calibration plot
📊 Calibration buckets (10 bins)
For each bin of predicted probability, the BLUE bar shows the actual win rate. The GREEN line marks where actual would equal predicted (perfect calibration). Closer the blue is to the green, the better.
📚 Reading this
Expected Calibration Error (ECE) = weighted average gap between predicted probability and actual win rate, across all bins. Lower is better:
  • ECE < 0.03 — excellent (publication-grade)
  • ECE 0.03 – 0.07 — good (production-ready)
  • ECE 0.07 – 0.15 — fair (model is biased)
  • ECE > 0.15 — poor (calibration needs work)
Bin gap direction:
  • If actual > predicted → brain is under-confident in this prob range. Predictions in this bin would be better calibrated if scaled UP.
  • If actual < predicted → brain is over-confident in this prob range. Scale DOWN.
The Calibrator and IsotonicCalibrator already auto-correct for these gaps — the diagram here shows the FINAL post-calibration result. If the high-prob bins (0.8+) are still showing big gaps, the brain has more learning to do on confident predictions.