Review

Conformal correlation

Perlo, Chiasserini, De Veciana and Malandrino apply ordinary Pearson correlation, the phi coefficient, to binary indicators of membership in conformal prediction sets, and read the result as a map of which classes a classifier confuses. The matrix is a descriptive summary of set co-membership, but conformal coverage supplies no inferential or semantic guarantee for its entries. Because the matrix discards the true labels and the prediction rankings, and can be driven by marginal inclusion rates and set size, it cannot in general identify directional class confusion or contain a superset of the confusion matrix. Eight worked examples show this by hand, and demonstration 32 shows it at scale.

What is proposed

The authors train an image classifier, hold out a labeled calibration set, and build adaptive prediction sets (APS) at miscoverage level \(\alpha\). For each test image they record an indicator vector \(z = (z_1, \dots, z_H)\), with \(z_h = 1\) when class \(h\) is in the set. The conformal correlation matrix (CCM) has entries

$$\rho_{ij} = \frac{\mathbb{E}[z_i z_j] - \mathbb{E}[z_i]\,\mathbb{E}[z_j]} {\sqrt{\operatorname{Var}(z_i)\,\operatorname{Var}(z_j)}}.$$

The authors read positive entries as classes the model confuses, negative entries as classes it keeps apart, and entries near zero as unrelated. They claim that the matrix sees potential errors where the confusion matrix sees only realized ones, that it can be estimated from unlabeled data, that it discloses less than a confusion matrix in federated training, and that it improves client selection when the goal is to separate two critical classes. The abstract describes it as capturing “the likelihood of class pairs co-occurring within conformal prediction sets.”

Worked examples

One fact organizes everything below. The conformal correlation matrix is computed from the sets alone, so it depends only on the distribution of the set \(C(X)\). Confusion depends on the joint distribution of three things: the set, the true label \(Y\), and the class \(\hat Y\) the model ranks first. Hold the distribution of the sets fixed and change \(Y\) or the ranking, and the matrix cannot notice. In every example the entries are computed from the definition,

$$\rho_{ij} = \frac{\mathbb{E}[z_i z_j] - p_i p_j}{\sqrt{p_i(1-p_i)\,p_j(1-p_j)}}, \qquad p_i = \mathbb{E}[z_i],$$

so each calculation needs only the inclusion rates \(p_i\) and the co-occurrence rates \(\mathbb{E}[z_i z_j]\). An entry is defined only for classes with \(0 < p_i < 1\), since otherwise a variance is zero, and the authors give no convention for that case. Each example prescribes the law of the sets directly. Nothing in the definition requires the sets to have come from any particular procedure, and we do not claim that adaptive prediction sets calibrated on a sample produce exactly these laws. Coverage of 100% in an example means only that every set contains the truth, so the marginal coverage lower bound is met. Example 1 is the exception. It builds two classifiers inside randomized APS exactly as Romano, Sesia and Candès specify it, and we lead with it. The rest are ordered with the simplest refutation and the most absurd result first.

Example 1. Opposite confusions, the same matrix, within randomized APS

Setup. Three classes with equal frequency. Two classifiers assign the probabilities \((0.6, 0.3, 0.1)\) to the classes in the orders shown, so the true class is always ranked second.

Calculation. The randomized APS score of a label is the probability mass at and above it in the ranking, less a uniform share \(u\) of its own mass. The true label is always second, so its score is \(0.9 - 0.3u\) under both models, uniform on \((0.6, 0.9)\). With the same calibration inputs and uniforms both models get the same threshold \(\tau\) in \((0.6, 0.9)\). Write \(r = (\tau - 0.6)/0.3\). At a test input the top class has score \(0.6 - 0.6U \le \tau\) and is always included. The second has score \(0.9 - 0.3U\), at most \(\tau\) exactly when \(U \ge (0.9 - \tau)/0.3\), an event of probability \(r\). The third has score \(1 - 0.1U > 0.9\) and is never included. Under either model each singleton \(\{A\}\), \(\{B\}\), \(\{C\}\) occurs with probability \((1-r)/3\) and each pair \(\{A,B\}\), \(\{B,C\}\), \(\{C,A\}\) with probability \(r/3\). Same law of sets, so same matrix, and the same coverage \(r\).

What it shows. Model 1 labels every \(A\) as \(C\), every \(B\) as \(A\) and every \(C\) as \(B\). Model 2 runs the cycle the other way. Their confusion matrices are transposes of each other, both with accuracy zero. The authors write that the matrix “contains a superset of the information conveyed by the confusion matrix.” It does not determine the confusion matrix even when the sets are produced by randomized APS exactly as Romano, Sesia and Candès specify it. Under the non-randomized score-threshold convention with every uniform set to zero, the pair survives too: every calibration score is 0.9, so the threshold is 0.9, the second-ranked class is always included, every set is the pair of the top-ranked and the true class, and the two models again share the law of sets with opposite confusions.

Example 2. A perfect classifier and a perfectly wrong one get the same matrix

Setup. This example makes the same point without the procedure, and with the accuracy gap as wide as it can be. Three classes \(A\), \(B\), \(C\) with equal numbers of images, and sets that always contain the true class and one neighbor.

Calculation. Each label is in two of the three sets, so every \(p_i = 2/3\) and every variance \(p_i(1-p_i)\) is \(2/9\). Each pair shares exactly one of the three sets, so every co-occurrence rate is \(1/3\). Every off diagonal entry is

$$\rho_{ij} = \frac{1/3 - (2/3)(2/3)}{2/9} = -\tfrac12.$$

The good model has accuracy 1. The bad model has accuracy 0. Coverage is 100% for both. The labels, the sets and therefore the matrix are identical for both.

What it shows. The authors write that the matrix “contains a superset of the information conveyed by the confusion matrix.” Here the confusion matrix has all its mass on the diagonal for one model and none there for the other, and the CCM is the same. The matrix never sees the ranking, so it cannot contain the confusion matrix.

Example 3. Zero correlation with 99% directional confusion

Setup. Three classes and a small number \(\varepsilon\), say \(0.01\). Images arrive in four kinds with the probabilities shown.

Calculation. \(A\) is in the sets of the first two kinds, so \(p_A = (1-\varepsilon)/2 + \varepsilon/2 = 1/2\). \(B\) is in the sets of the first and third kinds, so \(p_B = (1-\varepsilon)/2 + (1-\varepsilon)/2 = 1 - \varepsilon\). They appear together only in the first kind, so \(\mathbb{E}[z_A z_B] = (1-\varepsilon)/2\). That equals \(p_A p_B\), so the numerator of \(\rho_{AB}\) is zero and

$$\rho_{AB} = 0.$$

For indicators that are not constant, zero correlation is independence, so \(z_A\) and \(z_B\) are independent here. Yet among images whose true class is \(A\), the model ranks \(B\) first with probability \(1 - \varepsilon\), which is 99% at \(\varepsilon = 0.01\). Coverage is 100%.

What it shows. The authors' Remark 3 says that unrelated classes give an entry near zero. It is the converse that a reader of a heat map uses, that an entry near zero means the classes are unrelated, and the converse fails. Independence of set membership does not imply absence of classification confusion. The example also bears on their Remark 1 in its stated direction: the classifier exhibits 99% directional confusion from \(A\) to \(B\), and the entry is zero.

Example 4. Perfect correlation with one in a million co-occurrence

Setup. Every image has the same true class \(T\), which is in every set and ranked first, and \(A\) and \(B\) never occur as true labels. The labels \(A\) and \(B\) are added to the set together with probability \(\varepsilon\), and otherwise neither is added. Since \(T\) is in every set its own entries are undefined, and the claim concerns \(\rho_{AB}\).

Calculation. \(z_A = z_B\) on every image, so their correlation is

$$\rho_{AB} = 1 \quad \text{for every } 0 < \varepsilon < 1,$$

while the probability that \(A\) and \(B\) appear together in a set is \(\varepsilon\), which can be 0.4, or 0.01, or one in a million. Coverage and accuracy are 100%.

What it shows. The abstract says the matrix “captures the likelihood of class pairs co-occurring within conformal prediction sets.” The likelihood is \(\mathbb{E}[z_A z_B] = \varepsilon\). The entry is 1 regardless.

Example 5. Every pair becomes positively correlated from difficulty alone

Setup. Three balanced classes, with difficulty independent of the true class. Half the images are hard and get the set of all three classes. The other half are easy and get the singleton set of their true class. The true class is ranked first in both cases, so accuracy and coverage are 100%.

Calculation. Each label is in every hard set and in a third of the easy sets, so

$$p_i = \tfrac12 \cdot 1 + \tfrac12 \cdot \tfrac13 = \tfrac23, \qquad p_i(1-p_i) = \tfrac29.$$

Two labels appear together only in hard sets, so \(\mathbb{E}[z_i z_j] = 1/2\) for every pair, and

$$\rho_{ij} = \frac{1/2 - (2/3)(2/3)}{2/9} = \tfrac14.$$

What it shows. The authors read a positive entry as confusion between the pair, in their Benefit 1, which is the converse of their Remark 1. Here every pair is positively correlated, every entry is equal, and the classifier is perfect. The correlation comes from one shared cause across images, whether the image was hard, and owes nothing to any relation between the pair.

Example 6. Pooling two clients creates a strong correlation from nothing

Setup. Every image has the same true class \(T\), which is in every set and ranked first, and \(A\) and \(B\) never occur as true labels. On client 1 the labels \(A\) and \(B\) are each added to the set independently with probability 0.9. On client 2 they are each added independently with probability 0.1. The two clients contribute equally to the pooled data.

Calculation. Within each client \(z_A\) and \(z_B\) are independent, so \(\rho_{AB} = 0\) on client 1 and on client 2. Pooled, each label is in half the sets, \(p_A = p_B = (0.9 + 0.1)/2 = 1/2\), so each variance is \(1/4\). They appear together with probability

$$\mathbb{E}[z_A z_B] = \tfrac12 (0.9 \times 0.9) + \tfrac12 (0.1 \times 0.1) = 0.41,$$

and therefore

$$\rho_{AB} = \frac{0.41 - (1/2)(1/2)}{1/4} = 0.64.$$

Reverse the rates on one client, so that client 1 adds \(A\) with probability 0.9 and \(B\) with probability 0.1 and client 2 does the opposite. Then \(\mathbb{E}[z_A z_B] = \tfrac12(0.09) + \tfrac12(0.09) = 0.09\) and \(\rho_{AB} = (0.09 - 0.25)/0.25 = -0.64\).

What it shows. The authors compute matrices on a global calibration set across non identically distributed clients and read the entries as properties of the classifier. A strong positive or negative entry can be produced by the mixture of clients alone, with zero correlation on every client and no confusion anywhere.

Example 7. Two pairs the model never confuses get different entries

Setup. Three classes, cat, dog and plane, with equal numbers of test images. The classifier ranks the true class first on every image and returns the same set for every image of a given class.

Calculation. Every set contains the true class, so coverage is 100%. A third of the images are cats, so \(p_{\text{cat}} = 1/3\), and likewise \(p_{\text{dog}} = 2/3\) and \(p_{\text{plane}} = 1/3\). Each variance \(p(1-p)\) is \(2/9\). Cat and dog appear together on the cat images, so \(\mathbb{E}[z_{\text{cat}} z_{\text{dog}}] = 1/3\). Plane never appears with either, so the other two co-occurrence rates are zero. Substituting,

$$\rho_{\text{cat},\text{dog}} = \frac{1/3 - (1/3)(2/3)}{2/9} = \tfrac12, \qquad \rho_{\text{cat},\text{plane}} = \frac{0 - (1/3)(1/3)}{2/9} = -\tfrac12, \qquad \rho_{\text{dog},\text{plane}} = \frac{0 - (2/3)(1/3)}{2/9} = -1.$$

What it shows. The authors say the entries quantify “how easy to confuse those classes are,” and their Remark 2 attaches strong negative entries to dissimilar pairs. Read that way, dog and plane are strongly dissimilar and cat and plane less so. But the classifier treats plane identically with respect to both: it never puts plane in a set with either. The two entries differ only because dog is in twice as many sets as cat. For any two classes that never share a set, \(\rho_{ij} = -\sqrt{p_i p_j / ((1-p_i)(1-p_j))}\), a function of the inclusion rates and nothing else.

Example 8. Moving one label occurrence between rows moves an entry from 1 to almost 0

Setup. There are \(n\) test images, every image has the same true class \(T\), which is in every set and ranked first, and two rare labels \(A\) and \(B\) each appear in exactly one set.

Calculation. If they appear in the same set, \(z_A = z_B\) and \(\rho_{AB} = 1\). If instead the one occurrence of \(B\) is in a different set, then \(p_A = p_B = 1/n\), the co-occurrence rate is zero, and

$$\rho_{AB} = \frac{0 - 1/n^2}{(1/n)(1 - 1/n)} = -\frac{1}{n-1},$$

which at \(n = 1000\) is about \(-0.001\).

What it shows. The authors present heat maps of entries with no uncertainty attached. For rare labels moving a single label occurrence from one row to another moves an entry from 1 to about 0. That is half of the nominal range from \(-1\) to \(1\) and, by the floor formula above, the whole of the range attainable at these inclusion rates, from \(-1/(n-1)\) to \(1\).

What is computed

An APS set is built by sorting the softmax vector, cumulating it, and cutting the resulting rank dependent score at one threshold, with a randomization at the boundary. The threshold is the \((1-\alpha)\) quantile of the calibration scores, but any number in \([0,1]\) would produce a set of the same kind. Therefore everything in the matrix is a function of the probability vectors, one cutoff and the randomization. The probabilities already hold, at full resolution, what the indicators report at one bit per class.

The authors write that the entries “incorporate the entire distribution of predicted probabilities.” They incorporate one threshold of it. The sweep over four values of \(\alpha\), listed as a benefit, recovers coarsely what the threshold removed.

The natural comparator is the soft confusion matrix,

$$C_{ab} = \mathbb{E}\big[\,p_b \,\big|\, y = a\,\big],$$

the average probability the classifier assigns to class \(b\) over images whose true class is \(a\). It is directional and threshold free, and it needs no calibration set. It does need labeled evaluation data, as any confusion matrix does. It is a conditional mean, so it summarizes rather than determines: a model that puts probability 0.4 on \(b\) for every image of class \(a\) and a model that puts 0.6 or 0.2 with equal probability have the same entry and error rates of zero and one half. The confusion matrix of the ranked prediction carries that difference, so the two are complements. A comparator that needs no labeled evaluation sample is the mean pairwise product \(\mathbb{E}[p_i p_j]\), a threshold free soft association that is symmetric like the CCM. The authors also cite the coverage confusion matrix of Stutz et al., which stratifies by the true label and is directional. None of the three appears in an experiment. The only comparison offered is with the hard confusion matrix, so the contrast being drawn is soft versus hard rather than conformal versus not.

The certificate does not transfer

Split conformal coverage constrains one scalar,

$$\mathbb{P}(z_Y = 1) = \sum_y \mathbb{P}(Y = y,\; z_y = 1) \;\ge\; 1-\alpha,$$

averaged over exchangeable draws. It does not in general determine \(\mathbb{E}[z_i z_j]\), which mixes the true class with a wrong class, two wrong classes, and different true classes on different images, and it never justifies reading that rate as confusion. Coverage therefore supplies no calibration, no confidence interval and no interpretation for \(\rho_{ij}\). The calibration step chose a threshold, and that is the whole contribution of conformal prediction to the object. The authors do not say what else it would contribute.

The guarantee is also stated loosely. The authors write that “for a given input sample” the set “is guaranteed to contain the correct class” with probability close to \(1-\alpha\). Split conformal coverage is marginal. It does not imply coverage conditional on the input, on a class or on a client. In distribution-free regression, exact pointwise conditional coverage forces prediction intervals of infinite expected length at almost every nonatomic point (Barber, Candès, Ramdas and Tibshirani 2021, Proposition 1, restating Vovk 2012 and Lei and Wasserman 2014), and the ten client example below makes the classification point directly. The distinction matters here because the authors go on to interpret individual class pairs and individual federated clients as if validity transferred to them. The upper term is written as \(1/(n-1)\). For the randomized procedure with distinct scores the term is \(1/(n+1)\), so theirs is looser than necessary and probably a typo. Exchangeability, ties and the APS randomization are not stated.

The guarantee also does not transfer to a client or a class. Take ten equally weighted clients, nine with coverage 100% and one with coverage 0%. Pooled coverage is exactly 90%, which meets a nominal \(1-\alpha = 0.9\), and the tenth client has no protection at all. The same arithmetic applies to a class with 10% of the data that is missed every time. In the federated experiments the clients are non identically distributed by construction, and local matrices are compared with a global calibration set. Under that mismatch even the marginal guarantee need not hold, and it is not established. No coverage and no set sizes are reported for any client.

The entries are pinned by inclusion rates

Write \(p_i = \mathbb{E}[z_i]\) for the fraction of test sets containing class \(i\). Two classes that never share a set have \(\mathbb{E}[z_i z_j] = 0\), hence

$$\rho_{ij} = -\sqrt{\frac{p_i\, p_j}{(1-p_i)(1-p_j)}}.$$

With balanced inclusion rates and singleton sets this is exactly \(-1/(H-1)\).

The authors' Remark 2 says dissimilar classes are “expected to display strong negative correlation.” They display a number set by the marginal inclusion rates, which depend on the class frequencies, on \(\alpha\) and on the classifier. The same marginals bound the entries from above. The phi coefficient of two indicators cannot reach \(+1\) unless \(p_i = p_j\), so entries for pairs with different inclusion rates are not on a common scale. The abstract promised a likelihood of co-occurrence, which is \(\mathbb{E}[z_i z_j]\). The entry \(\rho_{ij}\) is a standardized excess co-occurrence whose attainable range depends on \(p_i\) and \(p_j\), so entries cannot be compared as an absolute scale of confusability. Example 7 above is the smallest case. Panel B of the demonstration draws the observed floor and the formula on top of each other.

Remark 1, that positive entries mark “intrinsically similar classes,” fails from the other side. A share of hard inputs, on which the classifier returns large sets, induces positive correlation between every pair of labels across images. Raise the share of hard inputs in panel B and all 45 pairs turn positive, including the 44 with no injected confusion. That is a set size effect, and the authors offer no adjustment for it. The statistic is also undefined whenever a class is always or never included, since its variance is zero, and no convention is given.

With the model, the calibration data and the auxiliary uniforms fixed the sets are nested in \(\alpha\), so as \(\alpha\) grows no inclusion rate increases and the zero co-occurrence floor moves toward zero or stays put, although nothing about the classifier has changed. The authors present the dependence on \(\alpha\) as a feature. Panel D shows the cost. As the sets approach singletons the matrix loses the ability to rank pairs by their injected confusion strength, while the soft statistic keeps it.

Symmetric object, directional claim

The matrix is symmetric, and it consults neither the true label nor the ranking. Example 1 gives two classifiers inside randomized APS with opposite confusions and the same matrix, and Example 2 does the same for a perfect and a perfectly wrong classifier. Example 3 gives a zero entry with 99% of one class labeled as the other. The authors nevertheless read from a positive entry that the classifier “is much more likely to misclassify images of class 91 as class 30 than as class 95.” Panel C of the demonstration slides the direction of a confusion from entirely 0 to 1 to entirely 1 to 0. The soft confusion entries cross, and the CCM entry is a curve symmetric about the midpoint.

Labels and privacy

The authors write that the matrix “can be estimated entirely from unlabeled samples.” The calculation ignores the test labels, but the threshold comes from a labeled calibration set. In the centralized experiment that set is a tenth of the data. For the federated experiment the calibration protocol at the client level is not described. The soft confusion matrix needs labeled evaluation data, as any confusion matrix does, and \(\mathbb{E}[p_i p_j]\) needs no labeled calibration or evaluation sample.

The privacy argument compares the CCM, which “only contains probabilities,” with a raw count confusion matrix, whose row sums reveal the size and class composition of a client’s data. Correlations can be negative, so the CCM does not contain probabilities at all. A row normalized confusion matrix does, and does not directly disclose counts. And when every set has at most one element and at least three classes have \(0 < p < 1\), the CCM identifies every inclusion rate exactly. Write \(o_i = p_i/(1-p_i)\). Then \(\rho_{ij} = -\sqrt{o_i o_j}\), so for any three distinct such classes

$$o_i = -\frac{\rho_{ij}\,\rho_{ik}}{\rho_{jk}}, \qquad p_i = \frac{o_i}{1 + o_i}.$$

Inside randomized APS this recovers a client's class proportions exactly when the classifier is perfect and every class has the same top probability. Take a perfect classifier that assigns the ranked probabilities \(0.6, 0.3, 0.1\) with the true class first, and class proportions \(\pi_1, \pi_2, \pi_3\). Every calibration score is \(0.6 - 0.6u\), so the threshold \(\tau\) is below 0.6, the top class is kept with probability \(r = \tau/0.6\), and no other class is ever included. The set is \(\{Y\}\) with probability \(r\) and empty otherwise, so \(p_i = r\pi_i\). The formula above recovers every \(p_i\), and then \(\pi_i = p_i / \sum_j p_j\). For an empirical matrix the same algebra recovers the empirical inclusion frequencies, from which the proportions are estimated rather than read off. There is no threat model, no privacy mechanism and no leakage bound. The defensible claim is that the CCM does not transmit raw counts.

The experiment

The client selection section contradicts the rest of the paper. In the centralized experiments a high entry means two classes are hard to distinguish, and the entry falling with training is read as the classifier learning to separate them. In the federated section the stated goal is “to make the corresponding CCMs to be as high as possible,” and the rule selects three clients per round from “the 7 clients with the highest CCM entries for the critical classes.” The rationale that follows is that a client “unable to distinguish the critical classes” is unlikely to help. By the authors’ own reading, the highest entries belong to exactly those clients.

The heterogeneity is generated with a Dirichlet whose concentration is described as producing more heterogeneous clients as it grows. For the symmetric Dirichlet it is the reverse. Figures 6 and 7 are captioned “CCM entries for the critical classes,” and their vertical axes are labeled class accuracy. The table reports the number of clients selected as six or seven, which the text does not explain. “Significant” is used throughout, and no repetitions, seeds, intervals or error bars are reported.

There is no baseline that could attribute anything to the conformal step. Random selection with the same pool size, selection by confusion matrix, by class counts, by local loss, by co-membership rate or by soft probability are all absent. A client whose data contains more of the two focus classes may have a larger local entry for that pair, so the rule may be selecting on class composition.

The conclusion says the method improves the critical classes “without hurting the overall classification performance” after summarizing the change as “small effects on the accuracy of non-critical classes” and then as “does not affect the performance for non-critical classes.” Table 1 says otherwise.

That is a redistribution of accuracy toward the weaker focus class, paid for by the stronger one and, on CIFAR-100, by the average over the other classes. It might be a defensible trade for an application that cares about one class above all others. The authors do not present it as a trade, and without repetitions they cannot say whether any of the differences would survive a second seed.

What a fair version would look like

If the goal is a picture of which classes the model cannot tell apart, use the probabilities. The soft confusion matrix is directional and threshold free. If labels are unavailable, \(\mathbb{E}[p_i p_j]\) is symmetric and needs no labeled calibration or evaluation sample. If sets are the object, because a downstream consumer receives sets and nothing else, then report the co-membership rate \(\mathbb{E}[z_i z_j]\) or the coverage confusion matrix of Stutz et al., and say plainly that the certificate covers the true class and nothing about the neighbors. In every case report the inclusion rates and set sizes alongside, since a phi coefficient cannot be read without them.

The habit

The authors did not stop at the conformal step. They went past it, into a second order statistic, which is the move this site argues for. However, they took the set as the object and threw away the probabilities to get it. The packaging teaches that the set is the certified thing, and so anything built on the set is assumed to inherit the certificate. What the set inherited was a threshold, chosen by a procedure whose one theorem concerns a quantity the matrix ignores.

The conformal step is a univariate transform. Used as a transform, its output is a coordinate that can be modeled further. Used as a coarsening, it is a lossy binarization with a certificate attached to the one bit that the downstream analysis then discards.