Calibration

do 70% odds actually happen 70% of the time? · How the app turns all of this into a grade

Anyone can say "70% chance." The only way to know whether that number means anything is to collect many such statements and check: of all the times the model said 70%, did the thing happen about 70% of the time? That check is called calibration, and it’s the difference between a model that quantifies uncertainty and one that decorates it. A weather forecaster who says 30% rain on 100 different days should get wet about 30 of them — more means underconfidence, fewer means overconfidence. Both are findings.

• Every daily ledger snapshot records the model’s own stated odds at that moment — the volatility model’s probability that each name is higher in about a month, and the card’s published "chance of reaching the analyst target within a year."
• When the horizon arrives, the outcome is scored against the recorded odds — recorded at the time, never backfilled, so the model can’t quietly rewrite its predictions.
• Results are grouped into bins (all the ~45% claims together, all the ~55% claims together) and the predicted column is put next to the realized column. They should match.
• The same ledger also checks rank quality: across each day’s names, did higher composites actually lead to higher returns? That correlation (the "information coefficient") is reported with its statistical strength — and split before/after the sector-profile change, so upgrades to the engine get their own verdict.

Whether to take the app’s probability language literally. If the 55% bin realizes at 55%, "55%" is information you can size positions with. If it realizes at 40%, the model is overconfident and you now know by how much. Either way you know something most tools never let you find out — because most tools never write their predictions down.

calibration: for each bin b,  realized(b) = wins(b) ÷ n(b)   — compare to predicted(b)
rank IC = Spearman correlation( composite ranks , forward-return ranks )  per date
t-stat = mean(IC) ÷ std(IC) × √n     (how unlikely this IC is if the true edge were zero)

bin

— a bucket of predictions with similar stated odds (e.g. all the 50–55% claims)

realized

— the fraction of those predictions that actually came true

IC

— information coefficient — rank correlation between scores and subsequent returns

t-stat

— roughly, how many standard errors the average IC sits above zero; ~2+ is meaningful

Calibration needs sample size — a bin with 12 observations proves nothing either way. The scorecard states every n and stays silent until the data can speak. And a well-calibrated model can still be unprofitable: calibration is honesty about uncertainty, not a promise of returns.