---
name: forecast-calibration
description: Log every probabilistic forecast, score resolved ones with the Brier score and log loss, and report calibration before claiming any edge.
---

# Forecast calibration scoring

Use this when you make probability forecasts (prediction markets, sales pipeline odds, launch-date estimates).

1. Log each forecast when you make it: id, question, resolution criteria, your probability p (0-1), timestamp, and the market price at that time if one exists. Never edit a logged forecast.
2. When a question resolves, record the outcome o (1 = happened, 0 = did not) from the official resolution source, with its link.
3. Score the resolved forecasts:
   - Brier score = mean of (p - o)^2. 0 is perfect; always answering 0.5 scores 0.25. Lower is better.
   - Log loss = -mean(o*ln(p) + (1-o)*ln(1-p)), with p clipped to [0.01, 0.99].
   - If market prices were logged, score the market the same way on the same questions. Your edge is your Brier minus the market's.
4. Calibration table: bucket forecasts by p (0-10%, 10-20%, ... 90-100%) and compare each bucket's average p with the share that actually happened.
5. Do not claim an edge (or ask to trade live) until you have enough resolved forecasts for the comparison to mean something. Your owner sets that minimum; report the count with every score.
6. Report at check-in: resolved count, your Brier, the market's Brier on the same set, and log loss, all computed from the log (cite the file).