Demonstration
The coverage lottery
You calibrate once. The stamp says 90%; a Beta draw decides what you got.
The marginal guarantee averages over two things at once. Marginal vs. conditional coverage unpacked the average over inputs. This page unpacks the other one: the average over calibration sets. You calibrate on the \(n\) points you happen to have, and from then on your band’s coverage is a fixed number. The theorem does not say that number is \(1-\alpha\). It says the procedure hits \(1-\alpha\) on average across calibration draws you did not get.
Conditional on your draw, the arithmetic is exact and old: realized coverage is the \(k\)-th order statistic of \(n\) uniforms, \(k=\lceil(n+1)(1-\alpha)\rceil\), so across draws it follows \(\mathrm{Beta}(k,\,n+1-k)\) whatever the score distribution (Vovk, 2012). At \(n=100\) and a stamped 90%, the middle 95% of draws spans roughly 84% to 95%. At \(n=10\), about one calibration set in five delivers less than 85% coverage, a case Kladny et al. (2025) press in medical imaging, where calibration sets that small are realistic and the patient gets one draw, not the average. Pinning realized coverage to within a point of the stamp, with 95% confidence, takes \(n\) in the thousands.
Each lab below calibrates once on its own \(n\) points. The histogram is what the labs actually got; the curve is the exact Beta law. Note what a bigger test set would buy: nothing. It only measures your fixed draw more precisely. The fixes are a bigger calibration set, or reporting the fan instead of the point.
The simulation draws each lab’s \(n\) scores and takes the \(k\)-th smallest; realized coverage is distribution-free, so uniform scores lose nothing. Push \(n\) down until \(k>n\) and the finite-sample correction demands the infinite band: 100% coverage, by vacuity. The conformal fan shows what dependence between test points does to this same picture.
Draw twenty points, then forty
The fan describes repeated draws, but a lab lives one path through it. Here eight labs each grow a single calibration set, twenty points, then forty, then eighty, recomputing their quantile at each step; press play to watch. Every path wanders inside the envelope, and the envelope narrows only like \(1/\sqrt{n}\): in the low hundreds the labs still disagree by several points about what their own stamped 90% delivers, and the middle half of draws remains more than a point wide at \(n=1{,}000\). Waiting a little does not settle the lottery; thousands of points do.
What the wobble costs: adverse selection
Suppose you treat the stamp as exact and quote it. You sell a contract paying \(\$1\) if your next band misses, for a premium of \(\alpha+s\). Averaged over calibration draws you are fine; that is the marginal guarantee restated as money. The trouble is who accepts. A counterparty who knows the residual distribution knows your band’s true miss rate \(p\), and trades only when your quantile came up short, \(p>\alpha+s\). The wobble is symmetric; the acceptance is not. Your expected concession per posted quote is \(\mathbb{E}\,(p-\alpha-s)^{+}\), and it shrinks only like \(\sqrt{\alpha(1-\alpha)/n}\): about 1.1 cents per dollar insured at \(n=100\), an 11% markup on the fair ten-cent premium, and it stays above a tenth of a cent until \(n\) passes ten thousand. Charging a spread \(s\) helps, but only by driving the informed flow away.
This levy is a different line item from the rent in the parimutuel note. That rent, \(I(R;X)\), is collected by a counterparty who conditions on \(x\), and it does not shrink as \(n\) grows. This one requires no side information beyond the marginal law of the residuals, is charged for the finite calibration set alone, and decays like \(1/\sqrt{n}\). A conformal quote can pay both at once.
Takeaway. The calibration size appears nowhere in the statement of the guarantee and everywhere in what the statement is worth. A stamped 90% from \(n=100\) is one ticket from a fan eleven points wide, and treating it as exact concedes a measurable edge to anyone who knows better. Before quoting the number, ask what your \(n\) makes of the fan.
Vovk (2012), Conditional validity of inductive conformal predictors, ACML (arXiv:1209.2673). Kladny, Schölkopf, Koch, Baumgartner & Muehlebach (2025), A critical perspective on finite sample conformal prediction theory in medical applications (arXiv:2512.14727).
Using conformal prediction in your own project? Tell Claude: “Read https://conformalprediction.net/SKILL.md and create a project skill from it.” It adds a check for whether your coverage is conditionally trustworthy.