📏 Probability Calibration
Make the model honest · Platt scaling so 70% confidence actually wins 70% of the time · --
🧪 Multi-Horizon
Why this matters: The raw logistic-regression probability is usually OFF. A model trained on imbalanced data or with regularization tends to be either overconfident (says 70% but actually wins 50%) or underconfident (says 55% but actually wins 70%). Every downstream decision (Trade of the Day score, position sizing, alert threshold) reads probability as if it were accurate — so if the raw prob is wrong, every decision is wrong.

How Platt scaling fixes it: We fit a sigmoid mapping P_cal = sigmoid(a × logit(P_raw) + b) to the observed (predicted, won?) pairs from every resolved prediction. The two parameters a (slope) and b (intercept) get fit by gradient descent on log-loss. After enough pairs (~30+), Calibrator.calibrate(rawProb) returns a honest probability that matches the realized win rate within sample noise.

Auto-update: Calibrator re-fits every 50 new resolved predictions or every hour, whichever comes first. No manual intervention needed.
Pairs recorded
0
Brier (raw)
—
Brier (calibrated)
—
ECE (calibrated)
—
📊 Reliability diagram
Each dot = one probability bucket. Perfect calibration = dots on the diagonal. Above the line = under-confident (model said 50%, actually won 70%). Below the line = overconfident.
🎯 Calibration mapping curve
What raw probabilities map to after calibration. The dashed line is identity (no change). The solid line is the fitted sigmoid.
📈 Calibration improvement
Improvement = (raw_metric - calibrated_metric) / raw_metric. Positive means calibration is helping.
⚙ Fitted parameters
a > 1 means model needs spreading out (was too soft). a < 1 means model needs softening (was too confident). b shifts the whole curve up/down.
BUCKET
N
AVG RAW
ACTUAL WIN%
AFTER CAL
GAP
🧪 Verification
You can prove calibration is working by:
1. After 30+ resolved predictions, click "Force re-fit" to compute initial parameters.
2. Watch Brier (calibrated) drop below Brier (raw) — that's the model becoming more accurate.
3. The reliability diagram dots should pull closer to the diagonal after re-fits.
4. Brain Monitor and TOTD pages can opt into calibrated probabilities by calling Calibrator.calibrate(rawProb).