Demonstration
The fence is the horizon
Marginal validity is cheap. It certifies the fence, not the cattle.
Split conformal delivers \(1-\alpha\) coverage and asks almost nothing of the predictor \(\hat\mu\). Here we take that generosity to its logical end. Three “predictors” run side by side, all at the same \(\alpha\) on the same data:
- A, good model + conformal. A degree-3 fit, score \(s_i=|y_i-\hat\mu(x_i)|\), band \(\hat\mu(x)\pm q\). Tight, ~\(1-\alpha\) coverage.
- B, garbage + conformal. Replace \(\hat\mu\) with a deliberately wrong constant. Same conformal machinery. The band balloons to absorb the error, yet coverage is still ~\(1-\alpha\).
- C, trivial randomized set. Ignore \(x\) entirely. For each test point emit the whole line with probability \(1-\alpha\), else the empty set. Coverage \(\to 1-\alpha\) by construction. The set conveys nothing.
Flip between them. Watch the coverage readout barely move while the band goes from a snug ribbon to a number that runs off the screen to, finally, the whole horizon flickering on and off.
Green = covered, red = missed. For C, “covered” means the prediction set was the whole line; “missed” means it was empty. Either way the set is the same regardless of where the point actually fell, it never looked at \(x\) or \(y\).
Conformal prediction “almost invites you to use garbage prediction functions.” A certificate that a fence contains the cattle 90% of the time tells you nothing if the fence is the horizon.
Takeaway. Validity is cheap. The same \(1-\alpha\) guarantee attaches to a careful fit, to a wrong constant, and to a coin flip over the whole real line. Coverage cannot tell a good predictor from a horizon-sized one, so a coverage number, alone, is not evidence that a method is any good. What would distinguish them is a score that punishes vagueness. Coverage ⊥ log-score shows the two pulling in different directions.