Weighted Interval Score (WIS)

Summary

The Weighted Interval Score (WIS) is a proper scoring rule used to evaluate probabilistic forecasts, particularly when they are represented as a set of predictive quantiles. WIS balances prediction accuracy (closeness of the predictive median to the observed value) with uncertainty calibration (how well the predictive intervals cover the observed outcomes). It is widely used in epidemiological and public health forecasting, including the CDC COVID-19 Forecast Hub.

Definition and Formula

For a single prediction interval defined by a lower bound $l$, an upper bound $u$, and an nominal coverage probability $(1 - \alpha)$, the Interval Score $\text{IS}{\alpha}$ for an observed value $y$ is defined as: $$\text{IS}(y > u)$$ where $\mathbb{I}(\cdot)$ is the indicator function. The three terms represent: 1. }(F, y) = (u - l) + \frac{2}{\alpha}(l - y)\mathbb{I}(y < l) + \frac{2}{\alpha}(y - u)\mathbb{ISharpness: The width of the predictive interval $(u - l)$. 2. Underprediction Penalty: A penalty proportional to how far the observation is below the lower bound. 3. Overprediction Penalty: A penalty proportional to how far the observation is above the upper bound.

To evaluate a full predictive distribution across $K$ intervals at nominal coverage levels $1 - \alpha_1, \dots, 1 - \alpha_K$ and the predictive median $m$ (corresponding to $\alpha_0 = 1$), the Weighted Interval Score is calculated as: $$\text{WIS}(F, y) = \frac{1}{K + 1/2} \left( w_0 |y - m| + \sum_{k=1}^K w_k \text{IS}_{\alpha_k}(F, y) \right)$$ where $w_k$ is a weight assigned to each interval (typically $w_k = \alpha_k / 2$ to make the score approximate the pinball loss or continuous ranked probability score).

Application in Epidemiology

In the COVID-19 Forecast Hub coordinated by the CDC, weekly probabilistic forecasts of hospitalizations and cases are submitted across 23 specified quantiles. The WIS is computed for each forecast-observation pair, allowing researchers to evaluate models on both their accuracy and their reliability in forecasting uncertainty. The Empirical Research Assistance (ERA) system was optimized against the WIS to produce its COVID-19 forecasting models.

Citations

• Bracher, J., Ray, E. L., Johansson, M. A., Apfel, D., Gerding, A., ... & Reich, N. G. (2021). Evaluating epidemic forecasts in an international comparison. PLoS Medicine, 18(2), e1003818. DOI: 10.1371/journal.pmed.1003818
• Aygün, E., et al. (2026). An AI system to help scientists write expert-level empirical software. Nature, 654, 909–915. DOI: 10.1038/s41586-026-10658-6. Source paper: s41586-026-10658-6.pdf