Demonstration
Crossing the gap, in the limit
Vovk’s universally consistent conformal predictive system: valid at every \(n\), sharp in the limit.
The information gap is a finite-sample statement about a family: a single residual shape, re-levelled to any coverage you like, concedes \(I(R;X)\) on the log score, and no amount of data changes that. This page runs the strongest counterpoint in the literature. Vovk’s universally consistent conformal predictive system (ALRW working paper 18) produces predictive distributions that converge weakly to the true conditional law for any iid data-generating distribution. The shape is made adaptive in the bluntest way possible: partition \(x\) into cells of width \(h_n\), a power of two, with \(h_n\to 0\) and \(n\,h_n\to\infty\), and let the predictive CDF at \(x\) be the within-cell empirical distribution of the labels, conformally calibrated. Lévy’s martingale convergence handles the shrinking cells; the law of large numbers fills them.
The first panel shows the predictive CDFs at a test point \(x^{*}\), against the true conditional law: the single-shape conformal CPD (one pooled residual shape around a fitted mean, the family the gap prices), the histogram system, and a two-stage parametric fit. Drag \(n\) upward and watch whose shape bends toward the truth.
The second panel is the race. Universal consistency is weak convergence, so each system is graded by its Wasserstein-1 distance to the true conditional law, averaged over \(x\); \(W_1\) metrizes weak convergence here and is well-defined for the step CDFs the histogram system outputs, so nothing needs to be smoothed into a density. The single-shape system plateaus at a positive level, the \(W_1\) shadow of the information gap. The histogram system descends, at roughly the rate its shrinking cells suggest. The parametric fit descends fast and then stops at its own bias floor: its cubic mean cannot represent the sine exactly, and the histogram system, which has no family to be wrong about, eventually passes it.
What runs here is the histogram form of the paper’s construction (Definitions 27–33); the fully conformal version of Theorem 31 differs in its randomized tie-handling and carries a stronger small-sample validity property. Note the coverage readout: throughout the race, at every \(n\), the histogram system’s 90% band covers at 90%. Validity never waited on consistency; sharpness is what arrives slowly. The cell-width multiplier shows the bind: wider cells fill faster but adapt less, narrower cells adapt faster but sit empty.
Isn’t this just a regression tree?
As an estimator, it is less than one: a regressogram, with cells fixed before the data arrive and no adaptive splits. The crudeness is load-bearing twice over. Fixed, label-blind cells keep the taxonomy equivariant, so the conformal validity argument survives untouched; a tree’s splits depend on the labels, and the exchangeability argument dies with them, short of sample splitting, which is a different construction. And nested dyadic cells make the shrinking-bandwidth step a Lévy martingale, which is what carries the consistency proof. So the theorem is about an intersection: exact finite-sample calibration and universal consistency can live in one object. It is not a claim that the object is a good estimator at your \(n\), and the race above shows that it is not. Read this way, the page confirms the accounting elsewhere rather than contesting it: crossing the gap is conditional distribution estimation by any name, here wearing its cheapest possible disguise, with the conformal layer contributing what it always contributes, the certificate.
Takeaway. The gap is a statement about a family, not a prohibition for all time. Make the shape adaptive and it can be crossed — in the limit, under iid, at nonparametric rates, with a bandwidth that must shrink and fill at once. At any finite \(n\) the question is only ever which estimator of the conditional law you are running — a plateau, a fast fit with a floor, or a slow one without — and the certificate is silent on that either way. The finite-sample accounting elsewhere on these pages is untouched; it lives at your \(n\), with your shape, and this page marks where it ends.
Vovk (2022). Universally consistent conformal predictive distributions. Pattern Recognition 126:108536; ALRW working paper 18 (arXiv:1708.01902, pdf). On the map it sits in the Cross region. Suggested in issue #120.
Using conformal prediction in your own project? Tell Claude: “Read https://conformalprediction.net/SKILL.md and create a project skill from it.” It adds a check for whether your coverage is conditionally trustworthy.