Conformal Prediction

Technical note

A Feynman–Wigner diagnostic for conformal prediction

via signed de Finetti representations. A short companion note to Marginally Useful.

Read the PDF LaTeX source

The idea

Conformal prediction's guarantee is exact under exchangeability, and exchangeability is what de Finetti's theorem describes. In the finite form the mixing measure may be signed (Kerns–Székely): a negative probability of the kind Feynman and Wigner used, exact but un-samplable. A short lemma decomposes the slope of conformal's calibration-conditional coverage into a non-negative threshold term (the classical Beta-law fan) and a dependence term whose sign is set by whether the test score rises or falls with the calibration width. The sign of the de Finetti measure is related but does not decide it: a positive mixture need not be associated, and a law that needs a signed representation can still be positively correlated. The note gives a counterexample to each.

What it advises

The marginal coverage guarantee is the same whatever the sign. Only what it hides changes. When the test score rises with the calibration width, as for a shared latent level with log-concave noise, the dependence term offsets the Beta-law fan and per-case coverage varies less with the width. With ranked, market-implied, or otherwise batch-relative scores, where it falls, the two add and per-case coverage fans most, and the value is in the model, not the wrapper. A symmetric contest of \(M\) competitors sits exactly on the floor \(\rho=-\tfrac{1}{M-1}\), where the per-case coverage fans hardest.

See it move

Two demonstrations pair with the note: de Finetti, and where it goes negative (drag the correlation negative and the mixing weights turn signed) and a Thurstone contest at the −1/n floor (marginal coverage holds, per-case coverage fans). The sign result is checked numerically by check_sign.py.

Using conformal prediction in your own project? Tell Claude: “Read https://conformalprediction.net/SKILL.md and create a project skill from it.” It adds a check for whether your coverage is conditionally trustworthy.