Technical note
The width of the conformal fan
Dependence and the variance of realized coverage. A companion note to the Feynman–Wigner diagnostic.
The idea
Fix the calibration set and the coverage you then realize is a random number, not \(1-\alpha\) exactly. For independent scores it follows the Beta\((k,n-k+1)\) law — the “fan,” with variance about \(\alpha(1-\alpha)/n\). This note shows the width of that fan is set by the sign of the cross-sample dependence. The mean moves with the dependence too (under independence it is the nominal \(k/(n+1)\); comonotone scores push it to \(\tfrac12\)), but the sharp story is the variance: to leading order it is the independent fan inflated by the factor \(1+(n-1)\bar c/(p(1-p))\), with \(\bar c\) the average pairwise covariance of the sub-threshold indicators at the operating quantile.
What the sign does
Positive (extendable) dependence adds a non-negative between-dataset term, exact through de Finetti and the law of total variance; the observed net effect is a wider fan. Negative dependence narrows it, to exactly zero at the maximally negatively associated contest floor \(\rho=-1/(n-1)\), where the realized coverage equals \(k/(n+1)\) on every draw. It is the same sign the companion note reads off the finite de Finetti measure: a genuine prior widens the fan, the signed corner collapses it.
What is proved, and what is left
The positive side is exact: a de Finetti / law-of-total-variance decomposition shows positive dependence adds a non-negative between-dataset term. The negative side is exact at both endpoints, along the whole iid-to-floor path (mixing in the contest keeps the mean fixed, so \(\operatorname{Var}=(1-t)\cdot\text{Beta}\)). The new result is exact summed over all levels:
\(\sum_k \operatorname{Var}(U_{(k)}) \le\) the independent value, for any negatively associated sample.
The proof uses the convex-order contraction of the sub-threshold count against a concave
functional, \(\min(j,\,n-N(u))\), the top-\(j\) sum: convex order reverses for concave maps, so the
expected-order-statistic profile majorizes the independent one, and Schur-convexity finishes it.
What about a single fixed \(k\)? The step behind one \(\operatorname{Var}(U_{(k)})\) is neither
concave nor convex, so the same order cannot sign it, and the single-level bound is
false for general negative association, at every \(n\). An explicit NA law with uniform
marginals — “at most one large”: at most one calibration score falls in a top bin
of width \(a\), the rest below — over-disperses its extreme order statistic,
\(\operatorname{Var}(U_{(n)})-\operatorname{Var}^{\mathrm{iid}}=c_n a^2+O(a^3)\) with \(c_n>0\) for
all \(n\) (\(c_2=\tfrac1{18}\), \(c_3=\tfrac3{40}\), \(\dots\); ratio \(1.0022\) at \(n=3\),
\(a=\tfrac1{20}\)). It is genuinely NA (single-trial multinomial cell structure), and it realizes
exactly the centroid-violating diagonal an LP predicts. The reason a single level escapes: the aggregate is controlled by the one-time law of the
count \(N(u)\), which negative association signs; a single level needs the two-time concordance of
\((N(u),N(v))\), which it does not. So per-level fan contraction is not a theorem of negative
association — and full support does not rescue it either: a constrained search yields a
strictly-positive-density (floor \(\tfrac15\)), strictly-interior-NA law whose
\(\operatorname{Var}(U_{(3)})\) still exceeds the benchmark by \(\approx1\%\), certified in exact
arithmetic. It is a theorem on the structured families a calibration design produces, which
we prove: the Farlie–Gumbel–Morgenstern copula
(\(\operatorname{Var}_\theta(U_{(k)})\) a downward parabola capped at the benchmark), sampling without
replacement, and on the Gaussian scale the exact monotone law
\(\operatorname{Var}(Z_{(k)})=v_k+\rho(1-v_k)\). All checks are reproduced by
check_fan_conjecture.py; a constructive corollary is that a pre-specified,
exchangeability-preserving randomized design reduces the run-to-run dispersion of realized coverage
without disturbing validity.
The time-series fan
When the calibration scores are a stationary segment of a series, the moments of realized coverage separate: the mean shifts by the Barber–Pananjady tax, at most \(\min_\tau\{\tau/(n+1)+2\beta(\tau)\}\) and vanishing in \(n\), while the dispersion inflates by the long-run-variance factor \(\sigma^2_{\mathrm{LR}}/(\alpha(1-\alpha))\) of the sub-threshold indicator process — a constant. Demo 22 shows both moments moving; the two-prices note adds the third charge, the sharpness rent.
See it move
An interactive companion, the conformal fan, lets you drag the cross-sample correlation from the floor to comonotone and watch the realized-coverage distribution narrow and widen, with the per-level and aggregate inequalities and the exact affine Gaussian law checked live against simulation.
Closest relative
Snell & Griffiths (2025), Conformal prediction as Bayesian quadrature (arXiv:2502.13228), is the loss-side generalization of the calibration-conditional law this note studies: Dirichlet(1,…,1) weights on the ordered calibration losses give a distribution-free posterior over realized risk, which collapses to the Beta fan under the 0–1 miscoverage loss. Conformal risk control is that posterior’s mean; their highest-posterior-density rule is the prescription “report the fan’s quantile, not its mean.” The guarantee is data-conditional: conditional on the calibration draw, this note’s axis. It moves along the fan, not across the information gap.
Using conformal prediction in your own project? Tell Claude: “Read https://conformalprediction.net/SKILL.md and create a project skill from it.” It adds a check for whether your coverage is conditionally trustworthy.