Demonstration 34 · Review

Set size as uncertainty

Ye, Yang, Pang, Wang, Wong, Yilmaz, Shi and Tu rank large language models by the mean size of their conformal prediction sets at nominal 90% coverage and read the size as the model’s uncertainty. Three experiments on a synthetic six-option classifier showing the mechanisms behind that. The first two are established results, due to Xi and co-authors and to Kostumov and co-authors, reproduced here in their simplest form: dividing the logits by a temperature changes no answer and changes the set size, and the two score functions the paper averages rank a sharp model and a soft model of equal accuracy in opposite orders. The third is structural and is the one that does not appear in the literature: two models with identical sets on every input have accuracy 90% and 5%. The close reading repeats the analysis on the paper’s own released logits.

Setup

Six options, as in the paper’s multiple-choice format. A synthetic model produces a logit vector for each question: it puts a margin \(a\) on one chosen option and Gaussian noise of scale \(\sigma\) on every option. The chosen option is the correct one with probability \(\text{acc}_0\) and otherwise a random wrong one, so accuracy is fixed by \(\text{acc}_0\) and does not depend on \(a\) or \(\sigma\). Probabilities are the softmax of the logits. The paper’s two set constructions, taken from its released code, are

$$\text{LAC:}\quad C(x) = \{\, y : \hat\pi_y(x) \ge 1 - \hat q \,\}, \qquad \text{APS:}\quad C(x) = \text{the top options in order, while the cumulative probability} \le \hat q,$$

with \(\hat q\) the \(\lceil (n+1)(1-\alpha) \rceil\)-th smallest calibration score, the score being \(1 - \hat\pi_y\) for LAC and the cumulative probability down to \(y\) for APS, and an empty set replaced by the top option. The set size statistic is the paper’s equation (6),

$$\mathrm{SS} = \frac{1}{|D_{\text{test}}|} \sum_{(X_t, Y_t) \in D_{\text{test}}} |C(X_t)|.$$

Each panel calibrates on 2,000 questions and evaluates on 6,000.

A. Temperature

Dividing every logit by \(T\) leaves every argmax where it was. Accuracy does not move. The APS set size does, by more than one option across the range at the defaults, and the LAC size moves less because LAC thresholds each option’s own probability while APS thresholds a cumulative sum that temperature reshapes. The dashed lines are realized coverage on the right scale. Raising the accuracy slider above 90% takes the LAC sets to singletons and the APS sets stay large.

Under the paper’s reading, the model at \(T = 0.5\) and the model at \(T = 2\) have different uncertainty. They answer every question identically. What the set size measures here is the scale of the logits.

B. Score function

Two models with the same accuracy, arranged by letting the chosen option beat the best other option by exactly the margin \(a\). The sharp model has a large margin and, when it is wrong, is confidently wrong. The soft model has a small margin. At the default \(\alpha = 0.15\), above the 10% of questions both models get wrong, LAC gives both models singleton sets and ranks them equal. APS gives the sharp model the larger sets, because APS admits options while the cumulative probability stays below \(\hat q\), and on a sharp model the options after the first carry so little mass that several fit under the threshold. The same two models, with the same answers on every question, are tied by one score and separated by the other. At \(\alpha = 0.10\), where the wrong fraction equals \(\alpha\), the LAC threshold sits on the boundary between the two blocks of calibration scores and the LAC ranking changes from one resample to the next.

The paper averages LAC and APS before ranking, and says so in order to “mitigate the influence of different score functions.” On its own released logits the two rankings agree with Kendall \(\tau\) between 0.58 and 0.88 across the five tasks, and two of its four headline examples of a more accurate model showing more uncertainty survive under APS only, with APS coverage of 95% and 99% on the more accurate model. The averaging did not remove the dependence.

C. Same sets, different accuracy

Eighty five percent of the questions are “medium”: the model puts probabilities \((0.5, 0.4, 0.1)\) on three options and zero on the rest. Model 1 puts the \(0.5\) on the correct option and model 2 puts the \(0.4\) on it. The remaining questions are “hard”: probabilities \((0.4, 0.3, 0.3)\) with the correct option in a uniformly random one of the three slots, identical under both models. The calibration scores under model 1 are \(0.5\) on medium questions and \(0.6\) or \(0.7\) on hard ones, and under model 2 they are \(0.6\) on medium questions and \(0.6\) or \(0.7\) on hard ones. The 90% threshold falls at \(\hat q = 0.6\) under both models whenever fewer than 10% of the calibration questions score \(0.7\), and at \(0.7\) under both otherwise. At \(\hat q = 0.6\) every medium question gets the set \(\{0.5, 0.4\}\) and every hard question gets its top option alone, under either model.

The sets agree on every question, so the set size and the coverage agree exactly. Model 1 is right 90% of the time and model 2 is right 5% of the time. The size statistic sees the two models as equally uncertain. The mean set size is a functional of the law of the sets,

$$\mathbb{E}|C(X)| = \sum_{y=1}^{K} \mathbb{P}\big(y \in C(X)\big),$$

and the no-go theorem for such functionals says that whenever a set of size two or more occurs, the label the model ranks first can be moved inside the set without changing any of them. Lowering \(\alpha\) enough moves the threshold to the medium block, and the two models then separate, because the score of the correct option differs. The paper’s ranking is at a single \(\alpha\).

The paper is Ye, Yang, Pang, Wang, Wong, Yilmaz, Shi and Tu, Benchmarking LLMs via Uncertainty Quantification, NeurIPS 2024 Datasets and Benchmarks, arXiv:2401.12794. The set constructions follow the paper’s released code. Source: js/demo34.js.