Demonstration 31
Two worlds
A pair of data generating processes that split conformal prediction cannot tell apart. The intervals agree, the marginal coverage agrees, and in one world the method is optimal while in the other it is leaving a measurable amount on the table.
World A. The forecast error is drawn from the same heavy tailed law every period, a Student \(t_\nu\). Large errors happen and they are bad luck. Nothing about this period made it dangerous, so one interval width is correct forever.
World B. Each period draws its own scale, \(\sigma^2 \sim \mathrm{InvGamma}(\nu/2, \nu/2)\), and given that scale the error is Gaussian. Large errors happen on high scale periods. If you knew the scale you would know in advance that the period was dangerous, so the correct interval is narrow when it is calm and wide when it is not.
These are not similar processes. They are marginally identical, because the Student \(t\) is exactly a Gaussian scale mixture. Split conformal only ever reads a quantile off the pooled residuals, so the calibration scores, the interval and the coverage agree in law between the two. The demo below goes further and hands both worlds the same calibration set, so the band is literally identical. No output of the method distinguishes them.
The upper panel draws the same band on both worlds. The lower panel is where they separate: conditional coverage against the percentile of the period’s scale. World A has no scale to condition on, so its curve is flat and sits on target. World B runs from over-covering the calm periods to badly under-covering the turbulent ones, crossing the target exactly once, and the certificate reports the same number throughout.
Push \(\nu\) upward and both worlds converge on the Gaussian: the scale mixture degenerates, the World B curve flattens onto World A’s, and the gap goes to zero. Pull \(\nu\) down toward 2 and the spread widens. Around \(\nu = 3\), a realistic tail index for daily financial returns, conditional coverage in World B runs from roughly 57% to 100% while the certificate says 90%.
Takeaway. The question “are the tails fat, or is the volatility stochastic?” is the standard disagreement about return series, and every practitioner has a view. Conformal prediction returns the same interval either way and certifies it either way. That is not a defect in the guarantee, which is exactly true in both worlds. It is what the guarantee is: a statement about the average over periods, which cannot distinguish a world where every period is alike from one where they differ and you could have known.
The information discarded by pooling is \(I(R;X) = h(t_\nu) - \tfrac12[\log(\nu/2) - \psi(\nu/2)] - \tfrac12\log 2\pi e\), shown in the readout. The closed form is checked against numerical integration and Monte Carlo in papers/matched-pair, and the identity behind it is in Marginally Useful. For why this is the same construction applied mathematics calls averaging out a fast variable, see homogenization.