David Freedman “Some issues in the foundation of statistics” Foundations of Science, vol. 1 (1995) pp.19–83. Reprinted in Some Issues in the Foundation of Statistics, Kluwer, Dordrecht (1997). Bas C. van Fraassen, ed.
… Statistical models originate in the study of games of chance, and have been successfully applied in the physical and life sciences. However, there are basic problems in applying the models to social phenomena; some of the difficulties will be pointed out. …
1. What is probability?
For applied workers, the definition [of probability] is less useful; [its defining characteristics] are not observed in nature. The issue is of a familiar type– What objects in the world correspond to probabilities? This question divides statisticians into two camps:
(i) the “objectivist” school, also called the “frequentists”;
(ii) the “subjectivist” school, also called the “Bayesians,” after the Reverend Thomas Bayes (England, c.1701-1761).
2. The objectivist position
Using Kolmogorov’s axioms (or more primitive definitions), we can construct statistical models that correspond to empirical phenomena; although verification of the correspondence is not the easiest of tasks.
3. The subjectivist position
… probabilities describe “degrees of belief.” There are two camps … , the “classical” and the “radical.” For a “classical” subjectivist, like Bayes … there are objective “parameters” which are unknown and to be estimated from the data. …
“Radical” subjectivists … deny the very existence of unknown parameters. For such statisticians, probabilities express degrees of belief about observables.
4. A critique of the subjectivist position
The subjectivist position seems to be internally consistent, and fairly immune to logical attack from the outside. … From an applied perspective, however, the subjectivist position is not free of difficulties. What are subjective degrees of belief, where do they come from, and why can they be quantified? No convincing answers have been produced.
My own experience suggests that neither decision-makers nor their statisticians do in fact have prior probabilities. A large part of Bayesian statistics is about what you would do if you had a prior.
Other arguments for the Bayesian position
… Now the argument shifts to the “normative:” if you were rational, you would obey the axioms, and be a Bayesian. This, however, assumes what must be proved. Why would a rational person obey those axioms? The axioms represent decision problems in schematic and highly stylized ways. Therefore, as I see it, the theory addresses only limited aspects of rationality. Some Bayesians have tried to win this argument on the cheap: to be rational is, by definition, to obey their axioms.
… taken as a whole, decision theory seems to have about the same connection to real decisions as war games played on a table do to real wars.
…. utilities may be just as problematic as priors.
In real examples, the existence of many states of nature must remain unsuspected. Only some acts can be contemplated; others are not imaginable until the moment of truth arrives. Of the acts that can be imagined, the decision-maker will have preferences between some pairs but not others. Too, common knowledge suggests that consequences are often quite different in the foreseeing and in the experiencing.
The fallback defense. Some Bayesians will concede much of what I
have said … . Still, the following sorts of arguments can be heard. The decision-maker must have some ideas about relative likelihoods for a few events; a prior probability can be made up to capture such intuitions, at least in gross outline. The details (for instance, that distributions are normal) can be chosen on the basis of convenience. A utility function can be put together using similar logic: … . The Bayesian engine can now be put to work, using such approximate priors and utilities. Even with these fairly crude approximations, Bayesian analysis is held to dominate other forms of inference … .
Here is my reaction to such arguments. Approximate Bayesian analysis may in principle be useful. That this mode of analysis dominates other forms of inference, however, seems quite debatable. … in real problems– where models and loss functions are mere approximations– the optimality of Bayes procedures cannot be a mathematical proposition. And empirical proof is conspicuously absent.
The rhetoric of “robustness” may suggest that such error analyses are routine. This is hardly the case even for the models. …
de Finetti. A beautiful theorem of de Finetti’s asserts that your opinion can be represented as coin tossing, the probability of heads being selected at random from a suitable prior distribution. This theorem is often said to “explain” subjective or objective probabilities, or justify one system in terms of the other.
Such claims cannot be right. … For example, suppose you have an exchangeable prior about those 0’s and 1’s. Before data collection starts, de Finetti will prove to you by pure mathematics that in your own opinion the relative frequency of 1’s among the first n observations will almost surely converge to a limit as n→∞. … What the theorem does is to show how various aspects of your prior opinion are related to each other. That is all the theorem can do, because the conditions of the theorem are conditions on the prior alone.
According to Jeffrey (1983, p.199), de Finetti’s result proves “your subjective probability measure [is] a certain mixture or weighted average of the various possible objective probability measures”– an unusually clear statement of the interpretation that I deny.
… the Bayesian approach can suggest statistical procedures whose behavior is worth investigating. But the theory is not a complete account of rationality, or even close. Nor is it the prescribed solution for any large number of problems in applied statistics, at least as I see matters.
5. Statistical models
In our days, serious arguments have been made from data. Beautiful, delicate theorems have been proved; although the connection with data analysis often remains to be established. And an enormous amount of fiction has been produced, masquerading as rigorous science.
I believe model validation to be a central issue. Of course, many of my colleagues will be found to disagree. For them, fitting models to data, computing standard errors, and performing significance tests is “informative,” even though the basic statistical assumptions (linearity, independence of errors, etc.) cannot be validated. This position seems indefensible, nor are the consequences trivial.
From Association to Causation: Some Remarks on the History of Statistics
… In my view, this modelling enterprise has not been successful. Investigators tend to neglect the difficulties in establishing causal relations, and the mathematical complexities obscure rather than clarify the assumptions on which the analysis is based.
Snow’s work on cholera, among other examples, shows that causal inferences can be drawn from non-experimental data. However, no mechanical rules can be laid down for making such inferences; since Hume’s day, that is almost a truism. Indeed, causal inference seems to require an enormous investment of skill, intelligence, and hard work. Many convergent lines of evidence must be developed. Natural variation needs to be identified and exploited. Data must be collected. Confounders need to be considered. Alternative explanations have to be exhaustively tested. Above all, the right question needs to be framed.
Statistical Assumptions as Empirical Commitments, Richard A. Berk, David A. Freedman, in Statistical Science, vol. 14 (1999) pp. 243–58. Reprinted in Journal de la Société Francaise de Statistique, vol. 140 (1999) pp. 5–32 and in Stochastic Musings: Perspectives from the Pioneers of the Late 20th Century. Lawrence Erlbaum Associates (2003) pp. 45–71. J. Panaretos, ed.
In the pages ahead, we will try to show how statistical and empirical concerns interact. The basic question will be this: what kinds of social processes are assumed by the application of conventional statistical techniques to convenience samples? Our answer will be that the assumptions are quite unrealistic. If so, probability calculations that depend on the assumptions must be viewed as unrealistic too.
… with respect to meta-analysis, our recommendation is simple: just say no. The suggested alternative is equally simple: read the papers, think about them, and summarize them.
Diagnostics Cannot Have Much Power Against General Alternatives, David A. Freedman Journal of Forecasting.
Model diagnostics are shown to have little power unless alternative hypotheses can be narrowly defined. For example, independence of observations cannot be tested against general forms of dependence. Thus, the basic assumptions in regression models cannot be inferred from the data. Equally, the proportionality assumption in proportional-hazards models is not testable. Specification error is a primary source of uncertainty in forecasting, and this uncertainty will be difficult to resolve without external calibration. Model-based causal inference is even more problematic.
Reference Guide on Statistics, david h. kaye and david a. freedman ,2nd ed. Federal Judicial Center, Washington, D.C. (2000)
Statistics is sometimes viewed, even by its practitioners, as a kind of craft skill whose validity depends on experience, not logic. Its computations are sometimes called ‘mathematical’, its analyses ‘rational’ and the results held to be ‘objective’. But in so far as statistics is genuinely, fully, mathematical, rational or objective, there must be some gap between its assumptions and what can be known about a domain of application. In effect, Freedman argues that this gap can matter, and unless given careful attention, significant errors can result. In this respect – contrary to the claims of some – statistics seems no different from other methods of analysis.