Review

Counterfactual online conformal under adaptive logging

Qiao, Lin, Ji, Wang and Yao study online conformal prediction when the prediction sets drive actions and only the chosen action’s outcome is seen. They blame the resulting miscoverage of rarely chosen actions on the actions shifting the outcome law, measure it by an “endogeneity coefficient”, and correct it with inverse-propensity weights. The problem is a real one and the paper states it clearly. Rerunning the paper’s own construction, we think the cause is better described as covariate shift in the mix of contexts the logger shows each action. The gap is unchanged when the action effect is set to zero and disappears when the logger ignores the context, and calibrating per context and action closes it without propensities. We also found that the weighted method often reaches its coverage by returning the whole real line for some action, which the paper’s set-quality figure does not show.

What is proposed

The paper is “Counterfactual Online Conformal Prediction Under Adaptive Logging,” arXiv:2609.30811, September 2026. There are finitely many actions. Each round a context \(X_t\) arrives, the learner issues one set per action, \(C_t(a)=\{y: s(X_t,y;a)\le \hat q_t(a)\}\), a logging policy picks \(a_t\), and only \(Y_t(a_t)\) is revealed. The threshold \(\hat q_t(a)\) depends on the action but not on the context. The target is counterfactual coverage,

$$\mathrm{CCov}_T(a) \;=\; \frac1T\sum_{t=1}^T \mathbf 1\{Y_t(a)\in C_t(a)\},$$

which averages over the marginal law of the context, including rounds where \(a\) was not played. Definition 2.5 measures the problem by the endogeneity coefficient

$$\rho \;=\; \sup_x \max_{a\ne a'} \mathrm{TV}\big(P(\cdot\mid x,a),\,P(\cdot\mid x,a')\big),$$

the largest difference, within one context, between the outcome laws of two actions. The text that follows says “The case \(\rho = 0\) recovers exchangeable online CP.” Theorem 3.2 builds a two-context, two-action instance on which per-action calibration on the logged scores misses counterfactual coverage by an amount of order \(\rho\). The fix, PW-OCP, runs the adaptive conformal recursion of Gibbs and Candès per action on the error signal \(Z_t(a)=\mathbf 1\{a_t=a\}\,\mathbf 1\{\text{miss}\}/\pi_t(a\mid X_t)\), which is unbiased for the counterfactual miss. A doubly robust variant adds an outcome model. The coverage rate is \((T\pi_{\min})^{-1/2}\), matched by a lower bound, and the abstract states that the methods “improve counterfactual coverage and downstream regret without sacrificing prediction-set sharpness.”

What the paper concedes, and what is already in print

Much of what a reader would raise, the paper raises itself. It needs positivity (“Positivity is essential”) and excludes logging that stops exploring. It notes that the weighted signal “becomes noisy when \(\pi_{\min}\) is small” and that it “requires the true logging propensity”. Its own sliding-window weighted quantile does about as well, and on regret “Online COPP gives the largest reduction.” On the Open Bandit and DJIA benchmarks plain ACI matches the new methods.

The ingredients are also in print, several without citation. That a counterfactual target and a logged sample “share the same conditional distribution \(P_{Y(1)|X}\) of the outcome but otherwise differ in the distribution of the covariates” is Lei and Candès (2021, Section 3.1), which the paper cites. Weighted conformal prediction under covariate shift is Tibshirani, Barber, Candès and Ramdas (2019), and conformal prediction when a model’s outputs choose the next data is Fannjiang et al. (2022). The doubly robust construction with “the product bias form” is Yang, Kuchibhotla and Tchetgen Tchetgen (2022). An importance-weighted miss indicator inside an online recursion is Wang, Zecchin and Simeone (2025), for intermittent feedback. None of these four is cited. Zheng and Jin (2026) argue that the per-action target itself “appears insufficient for decision-making”, and are not cited either.

The paper’s own contributions stand apart from these points: a clear statement of counterfactual coverage in the online setting, a telescoping identity for the weighted recursion that needs no projection, and a matching lower bound. What follows is not in any of the works above, as far as we can find, and we offer it as a suggestion for how the result might be framed.

The gap tracks the context mix rather than the action effect

The paper itself notes that action-conditional coverage “is therefore biased by the policy-induced context distribution,” and in our reading that is the whole mechanism. In the Theorem 3.2 construction the logger plays action 0 mostly in one context and action 1 mostly in the other. The threshold ignores the context while the score’s law depends on it, so the logged scores for an action come from the wrong mix of contexts. Whether the two actions’ outcome laws differ within a context, which is what \(\rho\) measures, does not seem to play a part.

Two variants of the paper’s variance-shift instance make this concrete. In the first, both actions have the same outcome law in each context, standard deviation 1 in one and 3 in the other, so \(\rho=0\) exactly, and the logger still favours one action per context. In the second the outcome laws differ as in the paper, so \(\rho>0\), but the logger ignores the context. The table reruns the paper’s setup: \(T = 20{,}000\), \(\alpha=0.1\), \(\varepsilon\)-greedy logging with \(\varepsilon = 0.05\), the paper’s ACI step of 0.005, PW-OCP at its rate-optimal step \(\sqrt{\pi_{\min}/T}\) with the iterate clipped to \([-0.5, 1.5]\) as in its appendix, ten seeds.

Worst counterfactual coverage gap over actions, \(\max_a|\mathrm{CCov}_T(a)-0.9|\), mean over ten seeds, computed on rounds 501 to 20,000 so that the start-up of every method is excluded. Standard errors are at most 0.003.

With \(\rho = 0\) the gap is 0.229, the same as the paper’s \(\rho>0\) instance. With \(\rho>0\) and a logger that ignores the context it is 0.002. So the statement that \(\rho=0\) recovers exchangeable online conformal prediction needs a qualification when the logger uses the context, as does the related-work claim that partial-feedback methods “assume per-action outcome laws are exogenous, i.e., \(\rho = 0\), so per-arm calibration on the logged subset is already unbiased.” Theorem 3.2 is an existence result and is not affected. Our suggestion is that the quantity indexing the failure is the dependence of the logger on the context together with the dependence of the score’s law on the context, rather than \(\rho\).

Conditioning on the context closes the gap without propensities

Counterfactual coverage averages over the marginal law of the context, and the shift is in the context. Coverage within each context therefore implies coverage under any reweighting of the contexts, which is the equivalence Gibbs, Cherian and Candès (2023) open with: “reformulating conditional coverage as coverage over a class of covariate shifts.” Run the same ACI recursion separately in each context and action, on the logged rounds only, with no weights. Every benchmark in the paper has a handful of discrete contexts: two, \(K\), and four aggregated contexts for the Open Bandit data. The last column of the table is this method. It matches PW-OCP on the counterfactual gap and needs neither positivity weights nor known propensities. The weighted approach remains the natural choice when contexts are many and propensities are known, and the two can be combined.

With a continuous context the per-cell route needs either bins or a score whose law does not depend on the context. Five bins already close the gap in the continuous case above, and the normalized score \(|y|/\sigma_a(x)\) with the true scale gives 0.003. A learned scale is not free, so the point has full force on the paper’s experiments and partial force in general.

How the weighted method reaches its coverage

PW-OCP’s threshold is still the empirical quantile of the logged scores, at level \(1-\alpha_t(a)\). For a rarely chosen action those scores come almost entirely from the contexts where the action does well, so reaching counterfactual coverage needs a level near 1, and the recursion repeatedly drives \(\alpha_t(a)\) to zero or below. The paper’s boundary convention (its eq. 6) then sets the threshold to \(+\infty\), so the set is the whole line. Its appendix clips the iterate to \([-0.5,1.5]\), which permits this.

Same runs as above. A round counts when the set for at least one action is the whole line. The per-cell method’s whole-line sets are early rounds of rarely observed cells.

The paper’s own Figure 4 is consistent with this: it shows the played action covered at 0.992 for PW-OCP against a target of 0.90. Its set-quality figure reports the threshold error “restricted to rounds with non-degenerate \(C_t(a)\),” so rounds with whole-line sets do not enter the evidence for “without sacrificing prediction-set sharpness.” The sharpness theorem, C.3, covers a projected recursion that keeps \(\alpha_t(a)\) inside \((0,1)\), and it would help to report the fraction of degenerate rounds alongside it. Our numbers come from a reimplementation, since the code is not yet public, so the exact fraction may differ from the authors’ runs.

The paper is Qiao, Lin, Ji, Wang and Yao, Counterfactual Online Conformal Prediction Under Adaptive Logging, arXiv:2609.30811v1. Quotations are from that version and its LaTeX source, and appendix settings from its Appendix F.1. Lei and Candès, Conformal inference of counterfactuals and individual treatment effects, JRSSB 2021, arXiv:2006.06138. Tibshirani, Barber, Candès and Ramdas, Conformal prediction under covariate shift, NeurIPS 2019, arXiv:1904.06019. Fannjiang, Bates, Angelopoulos, Listgarten and Jordan, Conformal prediction under feedback covariate shift for biomolecular design, PNAS 2022, arXiv:2202.03613. Gibbs, Cherian and Candès, Conformal prediction with conditional guarantees, arXiv:2305.12616. Yang, Kuchibhotla and Tchetgen Tchetgen, Doubly robust calibration of prediction sets under covariate shift, arXiv:2203.01761. Wang, Zecchin and Simeone, Mirror online conformal prediction with intermittent feedback, arXiv:2503.10345. Zheng and Jin, Prediction sets for counterfactual decisions, arXiv:2607.02206. The rerun is data/reviews/adlog_sim.py, with its output in adlog_sim.out.