Original paper

A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

Anastasios N. Angelopoulos · Stephen Bates

arXiv preprint · revised 2022 · 2021Read the paper ↗

A reading note on the work cited above. Explanations and reading questions are identified separately from the paper’s reported findings.

From a prediction to a useful set

A prediction set can be judged along two separate axes: how often it contains the outcome, and how much uncertainty it leaves unresolved. A set containing every possible answer succeeds on the first axis but offers little help. This elementary observation motivates reading coverage and informativeness together.

The split-conformal construction

Evaluate a fixed score on held-out calibration observations. A finite-sample corrected quantile determines which candidate responses enter the set. With exchangeable calibration and test observations, split conformal provides marginal coverage of at least 1−α, averaging over calibration and test randomness.

Marginal does not mean conditional

Barber and coauthors study the limits of distribution-free conditional inference. Their results clarify why a general marginal guarantee cannot simply be upgraded to useful exact coverage for every input without additional restrictions.

Hypothetical illustration: perfect coverage in a group containing 70% of cases, and two-thirds coverage in the remaining group, gives 90% overall coverage. The arithmetic establishes the aggregate rate while leaving unequal reliability between the groups visible.

Read the probability statement carefully

Suggested questions: which data fit the model, which calibrate the score, and which evaluate the final sets? Is coverage reported overall or within meaningful groups? Are widths reported alongside coverage? These checks help align a theorem with the decision a reader wants to make, without treating a nominal target as a guarantee for every individual outcome.

Sources & further reading

A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty QuantificationGentle Introduction · full text, §1.1, §3 and Appendix DThe limits of distribution-free conditional predictive inferenceConformalized Quantile RegressionConformal Prediction Under Covariate Shift

Continue reading

What generated scores tell us about uncertainty