Demonstration
How split conformal works
We build the method and watch it deliver exactly what it promises.
Split conformal prediction is simple and correct. Hold out a calibration set the model never trained on. For each calibration point compute a nonconformity score, here the absolute residual \(s_i = |y_i - \hat\mu(x_i)|\). Take the score’s \(\lceil (n+1)(1-\alpha)\rceil\)-th smallest value, call it \(q\), and emit the band $$C(x) = [\,\hat\mu(x) - q,\ \hat\mu(x) + q\,].$$ On fresh, exchangeable data this band covers the truth at least \(1-\alpha\) of the time. Move the sliders: the empirical coverage tracks the target. Nothing here is wrong. This is the part everyone agrees on.
Above: the calibration nonconformity scores \(|y-\hat\mu(x)|\). The dashed line is the conformal quantile \(q\); the band half-width is exactly that value. Coverage is achieved by counting, not by modeling.
Takeaway. It works, and it asks almost nothing of you, not even that \(\hat\mu\) be any good. That generosity is the first clue. In the fence-is-the-horizon demo we exploit it: the same 90% guarantee survives a deliberately terrible predictor. First, though, marginal vs. conditional coverage asks where that 90% actually lands.