Reviews
Reviews
Close readings of published papers. Each one asks what is proposed, what is computed once the vocabulary is stripped away, what the coverage certificate constrains, and what the experiments can support. Each is paired with a live demonstration that rebuilds the construction on a case where the truth is known.
The aim is to separate the part of a proposal that conformal prediction certifies from the part it only names. The two are routinely confused, and the confusion is easiest to see on a specific example.
Conformal correlation
Perlo, Chiasserini, De Veciana and Malandrino correlate class membership across conformal prediction sets and read the result as a map of which classes a classifier confuses. The statistic is the phi coefficient of prediction set membership indicators. Its negative entries are pinned by marginal inclusion rates, it is symmetric where confusion is directional, and the coverage certificate does not transfer to its entries.
Conformal correlation: the experiments
A synthetic ten class classifier with known confusability. The pinned floor and the formula drawn together, a confusion slid from one direction to the other under a symmetric statistic, and the loss of resolution against a probability based statistic as the sets shrink.
Conformal forecasting RNNs
Stankevičiūtė, Alaa and van der Schaar add one calibration quantile per horizon to an RNN point forecast and compare methods by joint coverage and width alone. That the COVID-19 guarantee is vacuous and the intervals inefficient was published by others in 2022 and 2024, and is set out here with sources. What appears to be new: the baselines are judged against a joint target they were never calibrated for, and under the proper score for an interval the quantile RNN the paper dismisses is the best interval on the paper’s own data on every split.
Constant-width intervals
The same point forecast with the CF-RNN width and with a width that follows a running scale, on a series with stochastic volatility. Same coverage, different interval score.
Relational conformal prediction
Cini, Jenkins, Mandic, Alippi and Bianchi train a graph quantile network on forecast residuals and read the interval off the fitted quantiles. That nothing is conformalized and that the intervals undercover throughout was said by the ICML reviewers and is set out here with sources. What appears to be new: the code calibrates by early stopping on the reported score, the synthetic experiment shows a residual forecaster rather than an uncertainty model, and on METR-LA a lookup table of conformal quantiles keyed by sensor, hour and current speed takes 85% of the distance from split conformal to the network, with no graph and nothing trained.
The missing step
Relational quantile regression on the residuals of a univariate forecaster, read directly and conformalized on held-out residuals, on a ring of coupled series with known truth. The direct read undercovers, the correction costs a few percent of score, and an oracle point forecast with one conformal quantile shows what the relational model is recovering.
Set size as uncertainty
Ye et al. rank language models by mean conformal set size. They concede that the size depends on the score function and average two scores to fix it. Rerun from their released logits, the fix fails on their own findings: two of the four headline examples in Section 6.3 survive under one score only, produced by that score covering at 95% and 99% rather than 90%.
Set size as uncertainty: the experiments
Temperature moves the set size at fixed accuracy, LAC and APS separate two equal-accuracy models differently, and two models with identical sets have accuracy 90% and 5%.
Certified uncertainty bounds
Karimi and Samavi certify per-input uncertainty bounds from a conformal set size by inserting the marginal coverage guarantee as if it held for each input. The bounds depend on the set size alone, and a two-group classifier with valid marginal coverage puts every input outside them, in a setting where calibrating per group would have given exact coverage.
Certified uncertainty bounds: the experiment
The two-group example with sliders: the certified interval is a line and both groups miss it.