English

Resolution of Simpson's paradox via the common cause principle

Methodology 2024-07-23 v2 Artificial Intelligence Probability Data Analysis, Statistics and Probability Applications

Abstract

Simpson's paradox is an obstacle to establishing a probabilistic association between two events a1a_1 and a2a_2, given the third (lurking) random variable BB. We focus on scenarios when the random variables AA (which combines a1a_1, a2a_2, and their complements) and BB have a common cause CC that need not be observed. Alternatively, we can assume that CC screens out AA from BB. For such cases, the correct association between a1a_1 and a2a_2 is to be defined via conditioning over CC. This setup generalizes the original Simpson's paradox: now its two contradicting options refer to two particular and different causes CC. We show that if BB and CC are binary and AA is quaternary (the minimal and the most widespread situation for the Simpson's paradox), the conditioning over any binary common cause CC establishes the same direction of association between a1a_1 and a2a_2 as the conditioning over BB in the original formulation of the paradox. Thus, for the minimal common cause, one should choose the option of Simpson's paradox that assumes conditioning over BB and not its marginalization. The same conclusion is reached when Simpson's paradox is formulated via 3 continuous Gaussian variables: within the minimal formulation of the paradox (3 scalar continuous variables A1A_1, A2A_2, and BB), one should choose the option with the conditioning over BB.

Keywords

Cite

@article{arxiv.2403.00957,
  title  = {Resolution of Simpson's paradox via the common cause principle},
  author = {A. Hovhannisyan and A. E. Allahverdyan},
  journal= {arXiv preprint arXiv:2403.00957},
  year   = {2024}
}

Comments

Added new results, enhanced references