Book Review: Bernoulli's Fallacy

Overview

Aubrey Clayton's Bernoulli's Fallacy is perhaps the most accessible introduction to Jaynes' version of "logical probability." But it goes further than that: Clayton provides the essential elements of the development and historical debates on the interpretation of probability and statistical inference through Fisher; gives numerous examples of how frequentist methods can fail; connects these issues to the academic research replication crisis; and suggests a way forward.

Interpreting probability

After motivating self-reflection within the reader through some example probability problems, Clayton enumerates the most common interpretations, including ancient notions of chance, frequentist, subjective, and axiomatic, while dissecting the appeal and problems of each one. Finally he presents "probability as logic," which will serve as author's point of view throughout the book.

To summarize, Kolmogorov's axiomatic probability had put probability on a rigorous foundation, but did not solve the interpretation problem; all modern interpretations could adhere to the axioms. The logical point of view then evolved with contributions from Keynes and Jeffries; Cox showed how to break probability free from the notion of proportions and frequencies and to represent plausibilities while still adhering to the axioms; and Jaynes completed the evolution with his book, Probability Theory: The Logic of Science. Clayton walks the reader quickly through this evolution, and the final intuition is as follows:

Probability is an extension of logical deduction. While logical deduction is process for calculating implications of interest from propositions that have true or false values, probability allows us to do the same with propositions of uncertain truth values. In fact, as Clayton relays, probability reduces to logical deduction when working exclusively with probabilities of 0s and 1s. Another key point is that all probability is conditional; we are always determining probability from some information, and that two rational people with the same information will calculate the same probabilities. Jaynes viewed probability as a way to process information, not as proportions or long run frequency. Finally, probability is epistemology, not ontology. That is, probability represents our state of knowledge rather than reality itself, and confusing our uncertainty with reality is the mind projection fallacy.

Following this, Clayton walks the reader through some classic probability problems using this view of the probability, pointing out along the way that there is no need in any of them to define probability as long run frequency.

Bernoulli's blunder

Clayton next turns our attention to the history of probability and statistical inference, starting with Bernoulli. In his telling of the history, Bernoulli was the first to make a mistake that would haunt statistics to the present day: concluding that a statistic derived from a particular sample is probably close to its true value for any sampled value, rather than the correct conclusion that a yet-to-be-drawn sample statistic will probably be close to the true value for any true value. Clayton explains that Bernoulli's fallacy came from confusing the sampling probability (the likelihood), with the probability of a hypothesis. These two are related through Bayes theorem, so Bernoulli needed to include more information to complete the calculation for his claim, and it's not necessarily true.

Clayton then goes on to illustrate why these two conclusions are not equivalent, despite appealing to people's intuition. Specifically, he arms the reader with a particular framework to define hypotheses, assign prior probabilities, assign the sampling probability (or likelihood), calculate the "pathway probability" (the unnormalized posterior of multiplying the previous two numbers), and finally the posterior probability for each hypothesis through normalization. This is done in a similar manner to Allen Downey's Bayesian worksheet (example use case here).

To end the chapter, Clayton covers base rate neglect, first presenting the canonical medical test example, followed by the prosecutor's fallacy with the tragic story of Sally Clark's conviction, before explaining that these are in fact the same error, confusing $P[A | B]$ for $P[B | A]$; the same one made by Bernoulli himself.

Wrong turn

The next chapter covers the bridge from applying statistics in the hard sciences to softer ones. This can be traced back to Gauss and Laplace, who developed and justified the method of least squares regression in astronomy, and Quetelet, who looked to apply the methods to the development of "social physics." Around this time, the common interpretation of probability also starts to shift toward frequency. This is due to both empirical data seeming to adhere to probability distributions, and because of academics questioning the origins of prior probabilities.

Chapter Four covers Galton, Pearson, and Fisher's influence on statistics. These three advanced frequentist statistical theory and applied it to comparisons of groups of people, for the purposes of eugenics. In this pursuit, they wanted to develop statistical methods that could be seen as "objective" to bolster their claims. Clayton wrote a related piece in 2020 that overlaps a bit with this chapter.

Chapter Five starts with an illustrative dialogue between a fictitious student and frequentist bot. The humorous conversation conveys all the typical confusions one encounters while trying to navigate the logic and interpretation of frequentist methods. It sits in stark contrast to the consistency of the previously presented Bayesian analyses.

Clayton then turns his attention back to Fisher, and the development of disparate tests for various situations, presenting a flow chart similar to the one presented by McElreath. But Clayton goes beyond this sight gag in his critique of frequentist methods, calling out issues in both theory and practice. He also cites evidence that Fisher, despite his vitriol toward Bayesian methods, actually drew inspiration from them in his derivation of maximum likelihood and grew a soft spot for them in his later writings. And finally, his zig zagging during the development of fiducial inference eventually led him to argue against his own earlier concepts of inference since the new paradigm would not work for his new paradigm, which was trying to draw Bayesian conclusions without use of priors.

The chapter concludes with nine example problems that require Bayesian reasoning to obtain sensible solutions and for which frequentist methods fail miserably.

Frequently wrong

Clayton then turns his attention to the replication crisis in academia, and particularly in the social and medical sciences. After citing objections to null hypothesis testing over time by those who recognized its irrelevance and misuse, he presents the theory and evidence of why so many modern studies have failed to replicate. Then he recounts the curious case of Daryl Bem, who turned from ESP skeptic to publishing "statistically significant" research on the subject. Bem's claims were implausible to most and yet used standard methods of analysis. Clayton then shares rumors that Bem had pulled this off as metacommentary on research practices.

In the final chapter, Clayton proposes reform to research practices. It will come as no surprise that the proposal includes abandoning frequentist methods, including null hypothesis testing, and instead embracing Bayesian methods. He even goes so far as to suggest that universities dissolve their statistics departments, since Bayesian inference relies on only one theorem. That seems a bit far to me, as Bayesian models and computation can become quite complex, and there are still regular developments in Bayesian methods and practice. Plus, even if researchers across fields used Bayesian methods, many won't master them at an advanced level, so it's helpful to have some centralized academics who can help or guide empirical work.

Recommendation

This is a unique book in its scope: a straightforward presentation of Logical Probability; an overview of the historical debates; hands-on examples of where frequentist methods fail and how Bayesian methods help; and insights and proposed remedies related to the academic replication crisis. It's not as math-heavy or dense as a textbook, but there is some math sprinkled throughout. The notation includes probability, summations, factorials, binomial coefficients, and exponential functions. So while I think the concepts are well-explained, a reader will want to feel comfortable with that notation and related concepts to get the most out of it.

Recommended for those looking for an introduction to Jaynes or the "statistics wars." I had first read this over two years ago, and it was fun to revisit and flip through. If you feel inspired to move on to Jaynes' book itself afterward, Clayton also has a video lecture series that covers the book from start to finish.