Demonstration 36 · Review

The missing step

Cini, Jenkins, Mandic, Alippi and Bianchi train a graph neural network on the residuals of a pre-trained point forecaster to predict the residual quantiles, and the interval is the point forecast plus the fitted \(\alpha/2\) and \(1-\alpha/2\) quantiles. The calibration set is the training set of the quantile model and nothing is held out to correct it. Here the construction is rebuilt on a ring of coupled series with known truth. The directly read quantiles undercover, the same regression conformalized on held-out residuals covers at a small cost in score, and the true point forecast with one split-conformal quantile per scaled residual does about as well. The close reading reruns the paper’s own METR-LA experiment with zero-parameter baselines.

Setup

Twenty-four series sit on a ring and each is hit one step later by its two neighbours’ shocks,

$$x_{i,t+1} = a\, x_{i,t} + \sigma_i\, \varepsilon_{i,t+1} + b\,\tfrac{1}{2}\big(\sigma_{i-1}\varepsilon_{i-1,t} + \sigma_{i+1}\varepsilon_{i+1,t}\big), \qquad a = 0.5,$$

with \(\varepsilon\) standard normal and the scales \(\sigma_i\) spread across series. The point forecaster is a pooled univariate autoregression of order two, fitted by least squares on a training slice. It never sees the neighbours, and their shocks cannot be recovered from its own past, so its residual \(r_{i,t+1} = x_{i,t+1} - \hat x_{i,t+1}\) has a mean that only the neighbours’ last residuals reveal. This is the situation the paper’s method is built for. Four intervals are built on the residuals of a calibration slice of \(n\) steps and scored on a long test stretch.

• SCP. One pooled split-conformal quantile of \(|r|\), the same half-width for every series and step.
• RelQR. Two linear quantile regressions at levels \(\alpha/2\) and \(1-\alpha/2\) on features observable at forecast time: the mean of the two neighbours’ last residuals, the series’ own last value and last residual, a running scale of its residuals, and a one-hot indicator of the series standing in for the paper’s node embeddings. Fitted by pinball loss on the whole calibration slice and read directly, which is the paper’s Eq. 14.
• RelQR+C. The same regressions fitted on the first half of the calibration slice, then corrected on the second half by the conformalized quantile regression score \(E = \max(\hat q_{\alpha/2} - r,\; r - \hat q_{1-\alpha/2})\): both ends move out by the \(\lceil (m+1)(1-\alpha)\rceil\)-th smallest \(E\).
• Oracle. The true conditional mean as point forecast, a running scale per series, and one split-conformal quantile of the scaled residual. This is what a base model that already knew the graph would give with plain split conformal, the paper’s own STGNN row.

Each interval is scored by

$$\mathrm{IS}_\alpha(l, u; y) = (u - l) + \frac{2}{\alpha}(l - y)_+ + \frac{2}{\alpha}(y - u)_+ ,$$

the interval score, which the paper calls the Winkler score. Lower is better.

At the default settings, sixty calibration steps of twenty-four series, the directly read regression lands a point or two under the target and the conformalized one lands on it, at a cost of a few percent of score. Over six resamples at the default the direct read covers between 86% and 90% and the conformalized read between 88% and 91%, with split conformal and the oracle at 90%. The shortfall of the direct read is modest here, which matches the paper’s own METR-LA column, and it grows as the calibration slice shrinks relative to the parameters: at thirty steps the direct read covers 85% to 88%, at ten steps under 80%. The correction does not restore the target on every draw. It is computed on the block of residuals that follows the block the regression was fitted on, and adjacent blocks of a time series are not exchangeable with the future, which is the very point the paper makes about its own residuals. The correction gets the average right where the direct read does not.

Setting the neighbour coupling to zero removes the predictable part of the residual, and the relational regressions have nothing left to learn beyond the per-series scale. Setting the scale heterogeneity to zero as well leaves all four intervals scoring alike, and the direct read still covers a point or two short: what it has fitted is noise.

The oracle row is the paper’s GPVAR finding made explicit. Their Table 2 shows CoRel on a univariate RNN reaching the width of split conformal on a graph forecaster, 1.33 against 1.32 with a theoretical optimum of 1.315. Here the relational regression closes about three quarters of the score gap between split conformal and the oracle, and the oracle is a point forecast that knows the neighbours plus one split-conformal quantile. The relational quantile model is recovering the term the point forecaster left out. That is useful when the base model cannot be retrained, and it is a forecast of the residual’s centre, not a statement about its spread.

The paper is Cini, Jenkins, Mandic, Alippi and Bianchi, Relational Conformal Prediction for Correlated Time Series, ICML 2025, arXiv:2502.09443. Conformalized quantile regression is Romano, Patterson and Candès (NeurIPS 2019). The interval score is Winkler (1972) and Gneiting and Raftery (2007), equation (43). Source: js/demo36.js.