- Testing hypotheses suggested by the data
In
statistics , hypotheses suggested by the data must be tested differently from hypotheses formed independently of the data.How to do it wrong
For example, suppose fifty different researchers, unaware of each other's work, run clinical trials to test whether Vitamin X is efficacious in treating cancer. Forty-nine of them find no significant differences between measurements done on patients who have taken Vitamin X and those who have taken a
placebo . The fiftieth study finds a big difference, but the difference is of a size that one would expect to see in about one of every fifty studies even if vitamin X has no effect at all, just due to chance (this could be caused bymeasurement error , regression to the mean, or just dumb luck that patients who were going to get better anyway ended up in the Vitamin X group instead of the control group.) When all fifty studies are pooled, one would say no effect of Vitamin X was found, because the positive result was not more frequent than chance, i.e. it was not statistically significant. However, it would be reasonable for the investigators running the fiftieth study to consider it likely that they have found an effect, at least until they learn of the other forty-nine studies. Now suppose that the one anomalous study was in Denmark. The data suggest a hypothesis that Vitamin X is more efficacious in Denmark than elsewhere. But Denmark was by chance the one-in-fifty in which an extreme value of the test statistic happened; one expects such extreme cases one time in fifty on average if no effect is present. It would therefore be fallacious to cite the data as serious evidence for this particular hypothesis suggested by the data.However, if another study is then done in Denmark and again finds a difference between the vitamin and the placebo, then the first study strengthens the case provided by the second study. Or, if a second "series" of studies is done on fifty countries, and Denmark stands out in the second study as well, the two series together constitute important evidence even though neither by itself is at all impressive.
The general problem
Testing an a hypothesis suggested by the data can very easily result in false positives (type I errors). If one looks long enough and in enough different places, eventually data can be found to support any hypothesis. Unfortunately, these positive data do not by themselves constitute evidence that the hypothesis is correct. The negative test data that were thrown out are just as important, because they give one an idea of how common the positive results are compared to chance. (An "effect" that occurs no more often than chance would predict is not an effect at all, much as a method for winning at gambling is only useful if it improves the odds over what the casino already gives you.) Running an experiment, seeing a pattern in the data, proposing a hypothesis from that pattern, then using the "same" experimental data as evidence for the new hypothesis is extremely suspect, because data from all other experiments has essentially been "thrown out" by choosing to look only at the experiments that suggested the new hypothesis in the first place. This is like turning over rocks in a forest which mostly have nothing under them, finally discovering a centipede under one particular rock, and saying that all rocks must have centipedes under them.
A large set of tests as described above greatly inflates the
probability oftype I error as all but the data most favorable to thehypothesis is discarded. This is a risk, not only in hypothesis testing but in allstatistical inference as it is often problematic to accurately describe the process that has been followed in searching and discardingdata . In other words one wants to keep all data (regardless of whether they tend to support or refute the hypothesis) from "good tests", but it is sometimes difficult to figure out what a "good test" is. It is a particular problem instatistical model ling, where many different models are rejected bytrial and error before publishing a result (see alsooverfitting ). Likelihood and Bayesian approaches are no less at risk owing to the difficulty in specifying thelikelihood function without an exact description of the search and discard process. The error is particularly prevalent indata mining andmachine learning . It also commonly occurs inacademic publishing where only reports of positive, rather than negative, results tend to be accepted, resulting in the effect known aspublication bias .How to do it right
All strategies for sound testing of hypotheses suggested by the data involve including a wider range of tests in an attempt to validate or refute the new hypothesis. These include:
*Collectingconfirmation sample s
*Cross-validation
*Methods of compensation formultiple comparisons
*Simulation studies including adequate representation of the multiple-testing actually involvedHenry Scheffé 's simultaneous test of all contrasts in multiple comparison problems is the most well-known remedy in the case ofanalysis of variance . It is a method designed for testing hypotheses suggested by the data while avoiding the fallacy described above. See his "A Method for Judging All Contrasts in the Analysis of Variance", "Biometrika ", 40, pages 87-104 (1953).ee also
*
Type I and type II errors
*Data-snooping bias
*Data analysis
*Exploratory data analysis
*Data dredging
*Predictive analytics
Wikimedia Foundation. 2010.