Demonstration 35 · Review

Constant-width intervals

Stankevičiūtė, Alaa and van der Schaar build multi-horizon forecast intervals by adding to a point forecast one calibration quantile per horizon. The half-width is the same for every input. They compare methods by joint coverage and mean width. Here the same point forecast gets two interval constructions on a series whose noise scale moves with time. One width for every state and a width that follows a running scale cover the same and are told apart by the interval score, the proper score for an interval. That these intervals are inefficient is established: Lin, Trivedi and Sun (TMLR 2022) and Sun and Yu (ICLR 2024) both say so, the latter cutting the width by a third with a copula in place of the Bonferroni correction. This panel isolates a different cause, holding the point forecast and the calibration fixed and varying only whether the width may depend on the state. The close reading runs the comparison on their COVID-19 data and on the FRED daily universe.

Setup

The series is an autoregression with stochastic volatility,

$$y_t = \phi\, y_{t-1} + s_t\, \varepsilon_t, \qquad \log s_t = \rho \log s_{t-1} + 0.8\sqrt{1-\rho^2}\,\eta_t,$$

with \(\varepsilon_t\) and \(\eta_t\) standard normal. The point forecast at horizon \(h\) from origin \(t\) is the exact conditional mean \(\hat y_{t+h} = \phi^h y_t\), so both interval constructions share the best available location. For each horizon a rolling window of the last \(W\) resolved absolute residuals \(|y_{o+h} - \hat y_{o+h}|\) is kept, and the CF-RNN construction is

$$\text{CF:}\quad \hat y_{t+h} \pm \hat\varepsilon_h, \qquad \hat\varepsilon_h = \text{the } \lceil (W+1)(1-a) \rceil\text{-th smallest residual in the window},$$

with \(a = \alpha/H\) for the Bonferroni intervals that target joint coverage \(1-\alpha\), and \(a = \alpha\) for the per-horizon intervals. The alternative keeps a running scale \(\hat\sigma_t\), the square root of an exponentially weighted mean of squared one-step residuals, divides each stored residual by the scale at its origin, and applies the same quantile to the scaled residuals,

$$\text{scaled:}\quad \hat y_{t+h} \pm \hat\sigma_t\, \hat\varepsilon^{\,\text{s}}_h, \qquad \hat\varepsilon^{\,\text{s}}_h = \text{the same quantile of } |y_{o+h} - \hat y_{o+h}| / \hat\sigma_o .$$

Both are split conformal quantiles over the same window at the same level. The only difference is whether the width may depend on the state. Each interval is scored at the level it was built for by

$$\mathrm{IS}_a(l, u; y) = (u - l) + \frac{2}{a}(l - y)_+ + \frac{2}{a}(y - u)_+ ,$$

which is proper for the pair of quantiles \((a/2, 1-a/2)\): width plus a penalty for each miss in proportion to how far the miss is, and no other construction of the interval has a smaller expected score. Lower is better.

Both constructions reach the target coverage. The constant-width interval is too wide in the quiet stretches and too narrow in the loud ones, and the interval score charges it for both. Setting the volatility persistence to zero makes the noise scale independent from step to step, the running scale has nothing to track, and the two constructions score alike. That is the case the paper’s static-noise synthetic experiments are built on. Its time-dependent noise grows with \(t\) in the same way for every sequence, which a running scale tracks as well.

The paper writes that “as long as the coverage rate surpasses target coverage, the intervals should be as efficient (narrow) as possible.” The interval score is that criterion made exact, and it ranks by the average of width and miss penalty rather than by the two numbers separately. A method that reaches the coverage target with a constant width has spent its whole budget on the average and has no way to be narrow where the state is calm.

The paper is Stankevičiūtė, Alaa and van der Schaar, Conformal Time-Series Forecasting, NeurIPS 2021. The interval score is Winkler (1972) and Gneiting and Raftery (2007), equation (43). Source: js/demo35.js.