Where it shines

Adaptive prediction sets

When the set is the deliverable, conformal prediction is exactly the right tool, and its size is an honest confidence signal.

Now the positive case. In classification, conformal prediction returns a set of labels guaranteed to contain the true one at least \(1-\alpha\) of the time. The score is \(s_i = 1 - \hat p_{y_i}(x_i)\), one minus the model’s probability on the true class; the set is \(C(x) = \{\,k : \hat p_k(x) \ge 1-q\,\}\). What makes this useful is that the set size adapts: a single label where one class clearly dominates, two or three where the input is genuinely ambiguous. That is precisely what you want for triage, return a confident answer when you can, a short ranked shortlist when you cannot, and let a human take the ambiguous ones. The coverage is the contract; the size is the “how sure am I.” Move overlap up and watch the sets grow where the classes collide.

The distribution of prediction-set sizes over the test inputs. Singletons (green) are the confident calls; size-2 and size-3 sets (amber, red) are where the model honestly reports ambiguity rather than guessing. Coverage holds across all of them.