Where it shines
Adaptive prediction sets
When the set is the deliverable, conformal prediction is exactly the right tool, and its size is a usable confidence signal.
Now the positive case. In classification, conformal prediction returns a set of labels guaranteed to contain the true one at least \(1-\alpha\) of the time. The score is \(s_i = 1 - \hat p_{y_i}(x_i)\), one minus the model’s probability on the true class; the set is \(C(x) = \{\,k : \hat p_k(x) \ge 1-q\,\}\). What makes this useful is that the set size adapts: a single label where one class clearly dominates, two or three where the input is genuinely ambiguous. That is precisely what you want for triage, return a confident answer when you can, a short ranked shortlist when you cannot, and let a human take the ambiguous ones. The coverage is the contract; the size is the “how sure am I.” Move overlap up and watch the sets grow where the classes collide.
The distribution of prediction-set sizes over the test inputs. Singletons (green) are the confident calls; size-2 and size-3 sets (amber, red) are where the model reports ambiguity rather than guessing. Coverage holds across all of them.
Takeaway. Here the marginal guarantee is not a consolation prize, it is the product: a set you can act on, with a coverage contract attached. The adaptivity that was a limitation in marginal vs. conditional coverage (coverage moving around with \(x\)) reappears here as a feature: the set size moves with difficulty, exactly as a triage signal should. Same theorem, opposite verdict, because the question changed.
Using conformal prediction in your own project? Tell Claude: “Read https://conformalprediction.net/SKILL.md and create a project skill from it.” It adds a check for whether your coverage is conditionally trustworthy.