FAQ

Objections & FAQ

Straight answers to the questions the site tends to raise.

What is conformal prediction best for?

That is for others to opine on, and largely outside the author’s expertise. Some effort has gone into collecting examples where coverage is genuinely the objective — prediction sets, anomaly detection, retrieval, robotics, language models, risk control — on the applications page, with the papers that do each well. Corrections and additions are welcome: pull requests gratefully received, especially nominations from practitioners who have shipped it.

Doesn’t conformalizing my model improve its uncertainty?

It gives the prediction set a coverage guarantee. It does not change your predictive distribution’s sharpness, and a proper score (log-likelihood, CRPS) is unmoved by the conformal step. If the model is overconfident, conformal widens the set to hit coverage, but the density it implies is just your model’s residual shape, re-levelled. Watch coverage and log-score decouple in the coverage ⊥ log-score demo, see the re-levelling side by side in conformal vs. recalibration, and read the Marginally Useful note for the exact size of what stays untouched: \(I(R;X)\).

But conformal predictive systems and CQR output whole distributions!

They do, and they can be excellent. The point is attribution: their sharpness comes from the conditional model they wrap, the quantile regressor, the difficulty estimator, the per-region bins, not from the conformal step, which supplies the coverage certificate. A worked example shows raw quantile regression matching its conformalized version, and the coverage-score plane plots where each method actually moves.

Can I get conditional (per-\(x\)) coverage?

Not exactly, and not for free: distribution-free, finite-sample conditional coverage forces infinite-length intervals (the price of conditional coverage; the full set of no-go theorems is the wall on the map). You can approach it under assumptions, conformalized quantile regression, normalized/Mondrian conformal, recent pivotal-score and optimal-transport methods, all of which condition on \(x\). The marginal-vs-conditional demo shows what the marginal average hides, and the subgroup demo shows the group-conditional version buying only a wider band.

Where does the guarantee actually come from?

Counting, not concentration. For any distinct scores, exactly \(\lceil(1-\alpha)(n+1)\rceil\) of the \(n+1\) placements of the label “test” point at covered scores; exchangeability just says your placement is an even draw. Spin the pointer, watch the balanced-placement demo traverse all \(n+1\) placements with the acceptance average within \(1/(2t)\) of the orbit level \(k/(n+1)\), or read the note on Steinitz balancing as the construction's exact dual. The guarantee also averages over calibration sets: your one calibration draw gives realized coverage distributed \(\mathrm{Beta}(k,\,n+1-k)\), which the coverage lottery and the Bayesian quadrature demo make explicit.

What about option pricing, or financial distributions generally?

This is where the coverage objective is most clearly the wrong one. An option price is a tail integral, \(\mathbb E[(S_T-K)^+]\), a functional of the density’s shape — and coverage is blind to shape: two forecasts with an identical 90% interval can price the same out-of-the-money contract orders of magnitude apart. Watch it happen. The target is the conditional density itself: recover the risk-neutral density from the option surface (Breeden–Litzenberger, 1978 and its successors), or model the conditional density directly and grade it with a proper score. Conformal can wrap either for a coverage certificate, but the tail shape that sets the price is the model’s work, not the wrapper’s; the parimutuel note makes the gap precise. The demo page carries the fuller reference trail.

Does it work for time series?

Better than the folklore says as regards moving sideways in the coverage / accuracy diagram. Exchangeability is not the right perspective: it is the assumption under which the guarantee is exact, not a switch that turns it off. On a stationary series whose dependence vanishes beyond a finite lag, plain split conformal provably loses at most (dependence range)/(calibration size) of its coverage, plus a mixing term when the dependence merely fades — watch it in the dependence-tax demo. What genuinely breaks a fixed band is drift (the drift demo); there the adaptive methods (ACI, conformal PID, EnbPI) recover a long-run average coverage and are well worth using — just don’t read them as a per-step guarantee. Dependence also carries a second, permanent bill: the sharpness rent priced in the two-prices note. See the coverage games and conditional coverage and sharpness.

Then what should I use to make forecasts better?

Model the conditional distribution (heteroscedastic models, quantile regression, mixture or flow densities) and recalibrate against a proper score. Then conformalize, if you also need a distribution-free coverage certificate. The order matters: estimate first, certify second. The alternatives page lists the workhorses (t-GARCH, laplace, pymc_forecast), demo 17 races one against a conformal wrap, and the recipes page has the audit code.

What is your overall recommendation?

Use conformal prediction whenever your loss depends on whether the truth lands in a set: it certifies coverage, distribution-free and finite-sample, and nothing else does that. When you need sharpness, model the conditional distribution and score it properly; no re-levelling of coverage crosses the information gap. In cartoon form:

The cartoon is deliberately contradictory: the rope bridge is a genuine crossing, the binned (“Mondrian”) conformal route in the style of Vovk that the price-of-conditional-coverage demo walks, and there are other paths too. But ask yourself whether you want to restrict yourself to methods with coverage guarantees as you go after sharp conditional predictions. It can feel a bit like climbing a mountain pass carrying your bike.

Who is the author?

Peter Cotton, who runs the skaters benchmarks and contributes to open source under the “microprediction” GitHub handle, and is newer to conformal prediction than to time series; weigh the advice here accordingly.

Using conformal prediction in your own project? Tell Claude: “Read https://conformalprediction.net/SKILL.md and create a project skill from it.” It adds a check for whether your coverage is conditionally trustworthy.