Demonstration
Drift & time series
Fixed split conformal is a straw man here. The adaptive methods are not, so judge everyone on a proper score.
The textbook coverage statement is proved under exchangeability: the calibration data and the future point can be reordered without changing the joint distribution. Stationary dependence alone turns out to be survivable: the coverage loss is provably bounded by (dependence range)/(calibration size) (see the coverage games). But the series here is worse than dependent: the level trends and occasionally jumps to a new regime. A fixed split-conformal band that freezes a single width does not weaken gracefully under that kind of drift; it simply stops covering.
The literature has strong online repairs. A fair test pits them not only against fixed split but against a plain probabilistic model, and scores all of them on a proper criterion, not marginal coverage alone (which several methods hit by construction). So below we forecast a nonstationary series \(y_t = \text{level}_t + \sigma_t\,\varepsilon_t\) with known, time-varying \(\sigma_t\), a trend, a regime jump at the orange line, and a high-volatility block afterwards. The point forecast \(\hat y_t\) is a trivial rolling mean. We calibrate on \([0,t_0]\) (the dashed line) and then run five methods online over \(t>t_0\), all consuming the same \(\hat y_t\):
- fixed split (CP), constant half-width \(q\) from the calibration residuals. The straw man, kept for contrast.
- ACI (Gibbs & Candès, 2021), online \(\alpha_t\), width from the \((1-\alpha_t)\) quantile of a trailing window of \(|y-\hat y|\), updated by \(\alpha_{t+1}=\operatorname{clip}\!\big(\alpha_t+\gamma(\alpha-\mathrm{err}_t)\big)\).
- conformal PID (after Angelopoulos, Candès & Tibshirani, 2023), trailing-quantile base radius plus an integral correction on the running coverage error.
- NexCP (weighted) (Barber et al., 2023), a recency-weighted quantile of trailing residuals, weights \(\rho^{\,\text{age}}\).
- EWMA-vol Gaussian, the yardstick: estimate the conditional sd by RiskMetrics EWMA of squared residuals (\(\lambda=0.94\)) and emit \(\hat y_t \pm z\,\sigma_t\). This models the spread directly, so it also returns a full predictive distribution.
Pick the method shown in the main panel; turn up drift and watch the green points go red for fixed split. The readouts report, for the selected method, marginal coverage, the worst-window coverage (the local collapse the marginal hides), and the interval (Winkler) score at level \(1-\alpha\), $$\mathrm{IS} = \frac{1}{T}\sum_{t>t_0}\Bigl[(\hat h_t-\hat\ell_t) + \tfrac{2}{\alpha}(\hat\ell_t-y_t)\,\mathbf 1\{y_t<\hat\ell_t\} + \tfrac{2}{\alpha}(y_t-\hat h_t)\,\mathbf 1\{y_t>\hat h_t\}\Bigr],$$ a proper score that rewards narrow bands and penalises misses, lower is better; \(T\) counts the evaluation steps and \([\hat\ell_t,\hat h_t]\) is that step’s interval.
The second panel overlays the rolling coverage (trailing 80 steps) of whichever methods you tick on, against the dashed target. This is where you see fixed split (blue) collapse after the regime jump while ACI, conformal PID and the EWMA model track \(1-\alpha\). They are doing real work.
Fixed split sits near target while the data are roughly stationary, then collapses once the trend and the volatility block bite. The adaptive conformal methods and the EWMA-vol model oscillate around target because they re-estimate the width from recent realized residuals. Note what stays true for all of them: the trailing window dips hard right after the jump. The long-run average recovers; the window you are standing in does not.
Takeaway. Two readings, both from the worked example. First, fixed split conformal really does collapse under drift (≈0.56 marginal, ≈0.17 worst-window). But ACI, conformal PID and NexCP genuinely recover long-run and marginal coverage; they are competitive. Second, the repair is conditional modeling, not the conformal step itself: a plain EWMA-volatility Gaussian matches the adaptive conformal methods on the interval score (≈6.9 vs ≈7.0, against an oracle ≈5.5) and returns a full predictive distribution. Where conformal helps under drift, it helps by adapting the width to recent residuals, local conditional modeling, with conformal supplying the coverage leveling on top. And none of these delivers per-step conditional coverage: even the oracle’s worst-window coverage is ~0.82. The repairs buy long-run/marginal coverage, never coverage now. The paper works through why. Conformal vs. recalibration asks whether plain probabilistic recalibration would have served you better all along.
Using conformal prediction in your own project? Tell Claude: “Read https://conformalprediction.net/SKILL.md and create a project skill from it.” It adds a check for whether your coverage is conditionally trustworthy.