Where it shines
Calibrated anomaly detection
A conformal \(p\)-value is a distribution-free hypothesis test, so you can cap the false-alarm rate.
This is conformal prediction at its strongest, working as a test rather than a forecaster. Score each point by how unusual it is against a clean reference set, then form the conformal \(p\)-value $$p(x) = \frac{1 + \#\{\,i : s_i \ge s(x)\,\}}{n+1}.$$ For a genuine inlier this \(p\)-value is (super-)uniform, so flagging whenever \(p \le \alpha\) controls the false-alarm rate at at most \(\alpha\) (exactly \(\lfloor\alpha(n+1)\rfloor/(n+1)\) for continuous scores), no matter what the inlier distribution is, with a finite calibration set, no training of a threshold. Below, each test point is plotted at its \(p\)-value; everything under the dashed line is flagged. The inliers’ \(p\)-values fill out the uniform histogram, and the false-alarm rate tracks \(\alpha\) as you drag it. Detection power, catching the real anomalies, is the score’s job, and improves as you push the anomalies further out.
The conformal \(p\)-values of the true inliers, which are (super-)uniform on \([0,1]\) (dashed line). That uniformity is the whole guarantee: the fraction landing below \(\alpha\) is at most \(\alpha\), and nearly \(\alpha\), so the false-alarm budget is honoured by construction.
Takeaway. Type-I error control is coverage, and coverage is what conformal prediction certifies, distribution-free and finite-sample. You set the false-alarm budget and it is kept regardless of the unknown normal distribution. The sharper the score, the more anomalies you catch within that budget, but the budget itself is guaranteed. This is the rare setting where the marginal guarantee is the entire point.
Using conformal prediction in your own project? Tell Claude: “Read https://conformalprediction.net/SKILL.md and create a project skill from it.” It adds a check for whether your coverage is conditionally trustworthy.