Review

Set size as uncertainty

Ye, Yang, Pang, Wang, Wong, Yilmaz, Shi and Tu benchmark large language models by the mean size of their conformal prediction sets at nominal 90% coverage, averaged over two score functions. That the size depends on the score function is known, and the paper knows it: its limitations section concedes that “varying conformal score functions may yield disparate estimations of uncertainty,” and it averages the two scores to “mitigate the influence.” Recomputing from the released logits shows the remedy does not work on the paper’s own results. Two of the four headline examples in its Section 6.3, the cases it offers of a more accurate model showing more uncertainty, exist under one score only, and are produced by that score realizing 95% and 99% coverage where the other realizes 91%. Averaging hid the dependence rather than removing it. Separately, a construction with identical sets under two models gives them accuracy 90% and 5%, which no functional of the sets can distinguish. Demonstration 34 shows each effect on a synthetic classifier.

What is proposed

The paper is Ye et al., “Benchmarking LLMs via Uncertainty Quantification,” NeurIPS 2024 Datasets and Benchmarks track, arXiv:2401.12794. Nine model families are evaluated on five multiple-choice tasks with six options each, 10,000 questions per task, half used for calibration and half for testing. Two set constructions from the split conformal literature are used, LAC and APS, each at \(\alpha = 0.1\). The paper reports accuracy, coverage, and the mean set size

$$\mathrm{SS} = \frac{1}{|D_{\text{test}}|} \sum_{(X_t, Y_t) \in D_{\text{test}}} |C(X_t)|,$$

“which is a primary metric for conformal prediction,” and ranks the models by it. Table 1 gives “the mean values of LAC and APS,” and Appendix C.10 says the analyses average over three prompting strategies as well. The paper argues that because coverage is at least 90%, “the size of the prediction set can serve as a reliable indicator of uncertainty,” and it draws three findings: more accurate models may be more uncertain, larger models may be more uncertain, and instruction tuning tends to increase uncertainty.

What has already been said

The dependence of set size on the score function is established, and the authors state it themselves. Their limitations section says that “the prediction sets generated by conformal prediction could be significantly influenced by the conformal score function utilized. Thus, for a specific LLM, varying conformal score functions may yield disparate estimations of uncertainty.” Section 6.1 gives the remedy: “By taking the average value of these two score functions, we aim to mitigate the influence of different score functions on the evaluation of uncertainty, thereby ensuring a more rigorous and reliable assessment.” Kostumov, Nutfullin, Pilipenko and Ilyushin, running the same two scores over vision-language models, report that “APS can also result in a notably distinct ranking of VLMs in terms of uncertainty compared to LAC” and that “the coverage rates of the APS method presented in Table IV for each model considered significantly exceed the 90% threshold on each dataset.”

The temperature effect is established twice over. Xi, Huang, Liu, Feng and Wei “empirically show that current confidence calibration methods (e.g., temperature scaling) typically lead to larger prediction sets in adaptive conformal prediction” and “prove that predictions with higher confidence result in smaller prediction sets on expectation.” Ye and co-authors run the ablation themselves in Appendix C.7, note that “the temperature does not affect accuracy,” and report the size as stable.

That the usefulness of a set is a property of the score rather than of the guarantee is in the standard introduction to the subject. Angelopoulos and Bates: “if the scores are random noise, then the sets will contain a random sample of the label space, where that random sample is large enough to provide valid marginal coverage. This illustrates an important underlying fact about conformal prediction: although the guarantee always holds, the usefulness of the prediction sets is primarily determined by the score function.”

What follows takes those as given and asks a narrower question the literature has not: whether averaging the two scores does what Section 6.1 says it does, on the findings the paper actually reports. It does not.

What the certificate constrains

Split conformal prediction with a score \(s\) and \(n\) calibration points gives, for an exchangeable test pair,

$$1-\alpha \;\le\; \mathbb{P}\big(Y \in C(X)\big) \;\le\; 1-\alpha + \frac{1}{n+1}.$$

That is the whole of the guarantee. It is one number, the average coverage, and it holds for every score function. The sets themselves, and therefore their sizes, depend on the score. The mean size is a sum of inclusion rates,

$$\mathbb{E}|C(X)| = \sum_{y=1}^{K} \mathbb{P}\big(y \in C(X)\big),$$

and the certificate constrains only the term in which \(y\) is the true label. The other terms are free, and a different score, a different logit scale, or a different amount of overshoot above \(1-\alpha\) changes them without touching the guarantee. The word “rigorous” in the paper is borrowed from the coverage statement and spent on the size.

Two consequences follow. The first is that a size comparison across models is a comparison at whatever coverage each model actually realized, which is at least 90% and can be much more. The second is that the mean size is a functional of the law of the sets alone, so the no-go theorem from our review of conformal correlation applies: whenever a set of size two or more occurs, the label the model ranks first can be moved within the set without changing any set, so the same sizes are compatible with different accuracies. Demonstration 34, panel C, does this with LAC sets: two models with the same set on every question, the same mean size and the same coverage, with accuracy 90% and 5%.

Reproduction from the released logits

The authors released the option logits of every model on every question, with the calibration and test split fixed by a seed in their script. We reran their set constructions from those files. The accuracies in their Table 1 reproduce to the second decimal and the set sizes to within \(0.1\), except for InternLM-7B, whose released logits give sizes about \(0.1\) smaller than the paper’s. Scripts are in data/reviews/.

The two score functions disagree

The averaging in Section 6.1 is a remedy for a disagreement, so the size of the disagreement matters. Kendall’s \(\tau\) between the LAC ranking and the APS ranking of the sixteen base models is \(0.63\) on question answering, \(0.58\) on reading comprehension, \(0.70\) on commonsense inference, \(0.88\) on dialogue response selection and \(0.83\) on document summarization. On reading comprehension the two scores order 25 of the 120 model pairs in opposite directions, and on question answering 22. Averaging two orderings that disagree on a fifth of all pairs does not produce an ordering free of the score. It produces a third ordering, which agrees with neither and inherits whichever score has the larger sets.

Realized coverage is not 90%

Kostumov and co-authors saw this on vision-language models, and it is on the paper’s own data too. Under LAC the realized coverage on the test half lies between 88.5% and 94.5% for every model and task. Under APS it reaches 97.6% on question answering, 99.6% on reading comprehension and 99.7% on commonsense inference, and it is highest on the most accurate models. APS scores the true label by the cumulative probability down to it, so a sharp model that is confidently wrong on a tenth of the questions has a tenth of its calibration scores near one, the threshold is near one, and nearly every option is admitted. The size of such a set is the size of a set at 99% coverage. The paper compares it with sets at 92% coverage and reads the difference as uncertainty.

The headline examples

Section 6.3 gives four examples in which “the LLM with much higher accuracy exhibits greater uncertainty compared to its counterpart.” Recomputed from the released logits, averaged over the three prompts as the paper does, and split by score function:

The Qwen-14B example is the clearest. Under LAC its sets are of size \(1.01\) at 91.4% coverage, the smallest of any model on the task, and Yi-6B’s are of size \(1.53\). The reported inversion comes entirely from APS, where Qwen-14B’s sets realize 99.2% coverage. The Qwen-7B example is the same: LAC ranks it as less uncertain than Llama-2-7B, and APS at 95.1% coverage reverses it. The document summarization example holds under both scores. The dialogue example does not reproduce from the released InternLM-7B logits under either score. Across all pairs of models within a task, the more accurate model has the larger average size in 66 of 600 cases. Under LAC alone it is 48, and under APS alone it is 110.

Temperature

Xi and co-authors established the effect and the paper ran its own ablation, so this is a check rather than a finding, and it comes out between them. Dividing the logits by \(T\) changes no answer and no accuracy. Between \(T = 0.5\) and \(T = 2\) the size ranking of the sixteen base models changes for six to nine of them on each task, which is more movement than Appendix C.7’s single-model stability result suggests. It does not overturn the rankings on its own. Fitting each model’s temperature by maximum likelihood on the calibration half and recomputing on the test half leaves the ranking nearly unchanged, with \(\tau \ge 0.92\) on every task. The score function and the coverage overshoot, not the temperature, are what move the paper’s rankings.

Accuracy already carries most of it

Under LAC the Spearman correlation between accuracy and set size across models is between \(-0.89\) and \(-0.99\) on every task. Under the paper’s average it is between \(-0.88\) and \(-0.94\). The new axis of evaluation is, to that extent, accuracy read off a different scale, and the part that is not accuracy is the APS overshoot.

What survives

At a fixed score function and a matched realized coverage, the mean set size measures how well the score orders the options. Sadinle, Lei and Wasserman show that the smallest sets at a given coverage come from thresholding the true conditional probabilities, so a smaller size at the same coverage means the score is closer to the truth in that ordering sense. That is a property of the model’s probabilities and a legitimate thing to report. It is not certified, it is not a measure of uncertainty separate from the score, and it does not see which option the model chose.

The paper is Ye, Yang, Pang, Wang, Wong, Yilmaz, Shi and Tu, Benchmarking LLMs via Uncertainty Quantification, NeurIPS 2024 Datasets and Benchmarks, arXiv:2401.12794v3, with code and logits at github.com/smartyfh/LLM-Uncertainty-Bench. Rerun scripts: data/reviews/ye_rankings.py, data/reviews/ye_table1.py and data/reviews/ye_temperature.py. The prior work is Kostumov, Nutfullin, Pilipenko and Ilyushin, Uncertainty-Aware Evaluation for Vision-Language Models (arXiv:2402.14418); Xi, Huang, Liu, Feng and Wei, Does confidence calibration improve conformal prediction? (arXiv:2402.04344); and Angelopoulos and Bates, A Gentle Introduction to Conformal Prediction (arXiv:2107.07511). The minimal-size result is Sadinle, Lei and Wasserman, Least ambiguous set-valued classifiers with bounded error levels, JASA 2019. Version 3 of the paper, used here, no longer contains the UAcc metric that earlier versions proposed.