Methodology & onboarding

Uncertainty, made explicit.

  1. 1. Define the question

    State mutually exclusive outcomes, units, close date, resolution date, source, criteria and assumptions before forecasting. Once a forecast is submitted the definition is locked; amendments are separate linked questions.

  2. 2. State a probability or interval

    60% is not a guarantee: roughly six successes in ten comparable trials is the interpretation. Choice probabilities sum to 100%. Numeric/date forecasts state low, median and high with explicit central interval meaning; dates are selected and displayed as calendar dates; time-of-day deadlines are explicitly UTC.

  3. 3. Show what changed

    Forecast revisions, reaffirmations, withdrawals, comments and factors carry time and reason. Group labels identify correlation; conditional questions do not assert joint distributions.

  4. 4. Combine transparently

    One latest, non-withdrawn forecast per human contributor. Choice forecasts use an unweighted arithmetic mean; dispersion is the first-option population standard deviation. Bounded histograms use an equal-weight mixture only when every latest active human contribution declares the same contiguous bin grid; incompatible grids, intervals and quantiles are not pooled. Bin support has left-inclusive, right-exclusive edges, with the final edge closed. No within-bin or tail shape is inferred. Numeric/date intervals and quantile curves otherwise remain individual. Quantiles require an ordered median and state their covered outer percentiles; tail shape is unspecified.

  5. 5. Resolve, then score

    An owner records outcome, source and reason after resolution date. Corrections append to history. Bespoke Brier v2 is half the sum of squared option errors, ranging 0–1, equivalent to standard binary (p−y)²; lower is better. Historical choice scores recalculate under this convention. Numeric/date intervals use coverage and absolute median error; quantiles also use mean pinball loss on comparable percentile grids. Histograms use bin Brier v1 only when the outcome lies inside declared support, compare only identical grids/support; outside-support outcomes are flagged and not scored. Rankings include only eligible resolved binary/choice questions, equal correlated-group weights, enrollment before start, in-period forecasts, minimum coverage and shared ties.

  6. 6. Check calibration and act

    Calibration compares forecast and observed frequencies in 20-percentage-point binary bins, using one mean prediction/outcome per question, with question and contribution counts. Small samples and correlated questions limit conclusions. Scenario margin = demand × price − costs, with independent explicit assumptions and an action threshold; this is not causal or AI analysis.

Try the fictional example →