Demonstration
The price of a dumb model
Same coverage certificate, real data, two estimators. One of them costs you.
The previous demo made a point in principle: the universally consistent construction is a regressogram, a deliberately crude estimator, and its crudeness is what lets the validity proof go through. This page puts a number on what that crudeness costs, on real data. The motorcycle-crash dataset (\(n=133\)) records head acceleration against time since impact, and its conditional spread swings from near-zero in the quiet approach to enormous through the crash: the canonical heteroscedastic testbed.
Two conditional-distribution estimators run on it. The regressogram bins time and uses the within-cell empirical distribution of acceleration, exactly the object from the previous page. The kernel-density estimate is the sensible alternative: a Nadaraya–Watson local mean and local scale, a smooth heteroscedastic fit. Both are then wrapped in the same conformal step (CQR-style), so both carry the exact distribution-free coverage certificate. That is the crux: the guarantee never required the crude model. Choosing it is a choice you pay for and did not have to make.
Grey are training points, black are held-out test points; the two bands are the conformalized 90% intervals. Read the coverage line first: both sit near the target, as the certificate promises. Then read the width line. The regressogram’s band is blocky and, in the tails where its cells starve and fall back to the global shape, far too wide; the kernel band breathes with the data. Scores are averaged over 40 random calibration/test splits, so the numbers are stable; the picture shows one split.
How much of this is just a badly chosen cell width? The second panel sweeps it. The regressogram’s held-out CRPS traces a shallow U — too fine and the cells are noisy, too coarse and they blur the crash — and the smooth estimator’s CRPS is the dashed line beneath. No cell width reaches it.
CRPS is a proper score, so this is a fair comparison of the whole predictive distribution, not just the interval. The gap between the regressogram’s best point and the smooth floor is the price, in the units the score is measured in: roughly a quarter more error at the best cell width, and the intervals a quarter wider, for the identical guarantee.
Takeaway. Coverage is orthogonal to sharpness, which cuts both ways: the certificate attaches to a good model and a crude one alike (the fence is the horizon), and precisely because of that, conformalizing does not oblige you to use the crude model. Here two estimators hold the same 90% guarantee and the regressogram is worse on every proper measure, at every cell width. If someone reaches for the deliberately dumb construction in order to get the guarantee, this is the bill: they could have conformalized the sensible model and kept the certificate for free. Estimate first, certify second.
Data: Silverman’s motorcycle dataset, via the MASS R package (Venables & Ripley). The smooth comparator is a Nadaraya–Watson conditional mean and scale; the regressogram is the histogram construction of demo 27. Both are wrapped with conformalized quantile regression (Romano, Patterson & Candès, 2019).
Using conformal prediction in your own project? Tell Claude: “Read https://conformalprediction.net/SKILL.md and create a project skill from it.” It adds a check for whether your coverage is conditionally trustworthy.