Natural experiment

A natural experiment is an observational study in which the assignment of treatments to subjects has been haphazard: That is, the assignment of treatments has been made "by nature", but not by experimenters. Thus, a natural experiment is not a controlled experiment. Natural experiments are most useful when there has been a clearly defined and large change in the treatment (or exposure) to a clearly defined subpopulation, so that changes in responses may be plausibly attributed to the change in treatments (or exposure).^[1]

Natural experiments are considered for study designs whenever controlled experimentation is difficult, such as in many problems in epidemiology and economics.^[1]

Original map by John Snow showing the clusters of cholera cases in the London epidemic of 1854

History

One of the most famous early natural experiments was the 1854 Broad Street cholera outbreak in London, England.

On 31 August 1854, a major outbreak of cholera struck Soho. Over the next three days 127 people near Broad Street died. By the end of the outbreak 616 people died. The physician John Snow identified the source of the outbreak as the nearest public water pump, which he identified using a map of deaths and illness.

In this example, Snow discovered a strong association between the use of the water and deaths and illnesses due to cholera. Snow found that the water company (the Southwark and Vauxhall Company) that supplied water to districts with high attack rates obtained the water from the Thames downstream from where raw sewage was discharged into the river. By contrast, districts that were supplied water by the Lambeth Company, which obtained water upstream from the points of sewage discharge, had low attack rates. Given the near-haphazard patchwork development of the water supply in mid-Nineteenth Century London, Snow viewed the developments as "an experiment...on the grandest scale."^[2] Of course, the exposure to the polluted water was not under the control of any scientist. Therefore, this exposure has been recognized as being a natural experiment.^[3][4]

Recent examples

Family size

An example of a natural experiment was discussed in Angrist and Evans (1998).^[5] The authors wish to estimate the effect of family size on the labor market outcomes of the mother. The correlations between family size and various outcomes do not tell us how family size causally affects labor market outcomes because both labor market outcomes and family size may be affected by unobserved variables such as preferences and because labor market outcomes may itself affect family size (called "reverse causality", for example, a woman may defer having a child if she gets a raise at work). The study notes that two-children families with either two boys or two girls are substantially more likely to have a third child than two-children families with one boy and one girl. The sex of the first two children, then, forms a natural experiment: it is as if an experimenter has randomly assigned some families to have two children and others to have three or more. The authors are then able to credibly estimate the causal effect of having a third child on labor market outcomes.

Game shows

Within economics, game shows are a frequently studied form of natural experiment. While game shows might seem as artificial contexts, they can be considered as natural experiment due to the fact that the context arises without interference of the scientist. Game shows have been used to study a wide range of different types of economic behavior, such as decision making under risk^[6] and cooperative behavior.^[7]

Smoking ban

An example of a natural experiment occurred in Helena, Montana during the six-month period from June 2002 to December 2002 when a smoking ban was in effect in all public spaces in Helena including bars and restaurants. Helena is geographically isolated and served by only one hospital. It was observed that the rate of heart attacks dropped by 60% while the smoking ban was in effect. Opponents of the law prevailed in getting the enforcement of the law suspended after six months, after which the rate of heart attacks went back up.^[8] Note, however, that while this may have been a good example of a natural experiment (called a case-crossover experiment, where the exposure is removed for a time period and then returned), it is also a good example of how confounding variables can result in faulty conclusions being made. For instance, many smoking ban-heart attack studies fail to indicate that heart attack rates were already on the decline before the smoking ban was in place, or fail to take into account seasonal fluxes in heart attacks (highest in the winter months and lowest in the summer). For the Helena study in particular, the claim that 40% of pre-ban heart attacks were caused by passive smoking is not believable, considering that only 10-15% of coronary heart disease cases are thought to be caused by active smoking.^[9]

See also

• Experiment

References

1. ^ ^{a b} DiNardo (2008)

   DiNardo, J. (2008). "Natural experiments and quasi-natural experiments". In Durlauf, Steven N.; Blume, Lawrence E. The New Palgrave Dictionary of Economics (Second ed.). Palgrave Macmillan. doi:10.1057/9780230226203.1162. http://www.dictionaryofeconomics.com/article?id=pde2008_N000142. 

2. ^ Snow, J. (1855). On the Mode of Communication of Cholera (2nd ed.). London: Churchill. Excerpted in MacMahon, B. & Pugh, T.F. (1970). Epidemiology. Boston: Little Brown.
3. ^ The 1854 cholera outbreak is the example of a natural experiment discussed often by David A. Freedman, e.g. in Statistical Models: Theory and Practice (Cambridge University Press) [1], chapter 1.3 (pages 6-9).
4. ^ Snow's studies of the pattern of the disease were convincing enough to persuade the local council to disable the well pump by removing its handle. After the handle of the well-pump was replaced, the incidence of new cases dropped. In stopping the use of water from the well-pump, the authorities did an uncontrolled experiment (without a control group) and without randomization.
5. ^ Angrist, J. & W. Evans (1998) Children and their parents' labor supply: Evidence from exogenous variation in family size," American Economic Review, 88(3), 450-77.
6. ^ Post, Van den Assem, Baltussen, and Thaler (March 2008). Deal or No Deal? Decision Making Under Risk in a Large-payoff Game Show. SSRN 636508. 
7. ^ van den Assem, van Dolder, and Thaler (April 2010). Split or Steal? Cooperative Behavior when the Stakes are Large. SSRN 1592456. 
8. ^ Sargent RP, Shepard RM, Glantz SA, Reduced incidence of admissions for myocardial infarction associated with public smoking ban: before and after study., British Medical Journal, vol. 328 pp. 977-980, 2004 [2]
9. ^ Snowdon, Chris. "The Myth of the Smoking Ban Miracle." Spiked, September 24, 2009.

• DiNardo, J. (2008). "Natural experiments and quasi-natural experiments". In Durlauf, Steven N.; Blume, Lawrence E. The New Palgrave Dictionary of Economics (Second ed.). Palgrave Macmillan. doi:10.1057/9780230226203.1162. http://www.dictionaryofeconomics.com/article?id=pde2008_N000142.