Galton's problem

Galton’s problem, named after Sir Francis Galton, is the problem of drawing inferences from cross-cultural data, due to the statistical phenomenon now called autocorrelation. The problem is now recognized as a general one that applies to all nonexperimental studies and to experimental design as well. It is most simply described as the problem of external dependencies in making statistical estimates when the elements sampled are not statistically independent.Asking two people in the same household whether they watch TV, for example, does not give you statistically independent answers. The sample size, "n", for independent observations in this case is one, not two. Once proper adjustments are made that deal with external dependencies, then the axioms of probability theory concerning statistical independence will apply. These axioms are important for deriving measures of variance, for example, or tests of statistical significance.

Origin

In 1889, Galton was present when Sir Edward Tylor presented a paper at the Royal Anthropological Institute. Tylor had compiled information on institutions of marriage and descent for 350 cultures and examined the correlations between these institutions and measures of societal complexity. Tylor interpreted his results as indications of a general evolutionary sequence, in which institutions change focus from the maternal line to the paternal line as societies become increasingly complex. Galton disagreed, pointing out that similarity between cultures could be due to borrowing, could be due to common descent, or could be due to evolutionary development; he maintained that without controlling for borrowing and common descent one cannot make valid inferences regarding evolutionary development. Galton’s critique has become the eponymous "Galton’s Problem" (Stocking 1968: 175), as named by Raoul Naroll, [Cite journal
author = Raoul Naroll
year = 1961
title = Two solutions to Galton’s Problem
journal = Philosophy of Science
volume = 28
pages = 15–29
doi = 10.1086/287778] [Cite journal
author = Raoul Naroll
year = 1965
title = Galton's problem: The logic of cross cultural research
journal = Social Research
volume = 32
pages = 428–451] who proposed the first statistical solutions.

By the early 20th century unilineal evolutionism was abandoned and along with it the drawing of direct inferences from correlations to evolutionary sequences. Galton's criticisms proved equally valid, however, for inferring functional relations from correlations. The problem of autocorrelation remained.

olutions

Statistician William S. Gosset in 1914 developed methods of eliminating spurious correlation due to how position in time or space affects similarities. Today’s election polls have a similar problem: the closer the poll to the election, the less individuals make up their mind independently, and the greater the unreliability of the polling results, especially the margin of error or confidence limits. The effective "n" of independent cases from their sample drops as the election nears. Statistical significance falls with lower effective sample size.

The problem pops up in sample surveys when sociologists want to reduce the travel time to do their interviews, and hence they divide their population into local clusters and sample the clusters randomly, then sample again within the clusters. If they interview "n" people in clusters of size "m" the effective sample size (efs) would have a lower limit of 1 + ("n"-1)/"m" if everyone in each cluster were identical. When there are only partial similarities within clusters, the "m" in this formula has to be lowered accordingly. A formula of this sort is 1 + d ("n"-1) where d is the intraclass correlation for the statistic in question [ [http://faculty.smu.edu/slstokes/stat3370/deff.pdf Sample Size and Design Effect] ] . In general, estimations of the appropriate efs depends on the statistic estimated, as for example, mean, chi-square, r, regression coefficient, and their variances.

For cross-cultural studies, Murdock and White [Cite journal
author = George P. Murdock and Douglas R. White
year = 1969
title = Standard cross-cultural sample
journal = Ethnology
volume = 9
pages = 329–369.
url = http://repositories.cdlib.org/imbs/socdyn/wp/Standard_Cross-Cultural_Sample] estimated the size of patches of similarities in their sample of 186 societies. The four variables they tested – language, economy, political integration, and descent – had patches of similarities that varied from size three to size ten. A very crude rule of thumb might be to divide the square root of the similarity-patch sizes into "n", so that the effective sample sizes are 58 and 107 for these patches, respectively. Again, statistical significance falls with lower effective sample size.

In modern analysis spatial lags have been modelled in order to estimate the degree of globalization on modern societies (Jahn 2006).

Spatial dependency or auto-correlation is a fundamental concept in geography. Methods developed by geographers that measure and control for spatial autocorrelation (e.g., Cliff and Ord 1973, 1981) do far more than reduce the effective "n" for tests of significance of a correlation. One example is the complicated hypothesis that “the presence of gambling in a society is directly proportional to the presence of a commercial money and to the presence of considerable socioeconomic differences and is inversely related to whether or not the society is a nomadic herding society.” Tests of this hypothesis in a sample of 60 societies failed to reject the null hypothesis. Autocorrelation analysis, however, showed a significant effect of socioeconomic differences. [Cite journal
author = Malcolm M. Dow, Michael L. Burton, Douglas R. White, and Karl P. Reitz
year = 1984
title = Galton's Problem as Network Autocorrelation
journal = American Ethnologist
volume = 11
issue = 4
pages = 754–770
doi = 10.1525/ae.1984.11.4.02a00080]

How prevalent is autocorrelation among the variables studied in cross-cultural research? A test by Anthon Eff on 1700 variables in the cumulative database for the Standard Cross-Cultural Sample, published in [http://www.worldcultures.org/ World Cultures] , measured Moran’s I for spatial autocorrelation (distance), linguistic autocorrelation (common descent), and autoccorrelation in cultural complexity (mainline evolution). "The results suggest that ... it would be prudent to test for spatial and phylogenetic autoccorrelation when conducting regression analyses with the Standard Cross-Cultural Sample." [Cite journal
author = E. Anthon Eff
year = 2004
url = http://www.mtsu.edu/~eaeff/downloads/SAS2001.pdf
title = Does Mr. Galton still have a Problem? Autocorrelation in the Standard Cross-Cultural Sample
journal = World Cultures
volume = 15
issue = 2
pages = 153–170
note = Southern Anthropological Association conference paper, Nashville, TN] The use of autocorrelation tests in exploratory data analysis is illustrated, showing how all variables in a given study can be evaluated for nonindependence of cases in terms of distance, language, and cultural complexity. The methods for estimating these autocorrelation effects are then explained and illustrated for ordinary least squares regression using again the Moran I significance measure of autocorrelation.

Anthon Eff explains how, when autocorrelation is present, it can often be removed to get proper estimates of regression coefficients and their variances by constructing a respecified dependent variable that is "lagged" by weightings on the dependent variable on other locations, where the weights are degree of relationship. Ordinary least squares regression will still bias the estimated coefficients when the dependent variable is respecified, but that maximum likelihood methods (Anselin 1988) will give unbiased statistics and variances in which the effects of autocorrelation have been removed.

The Durbin–Watson test [Douglas R. White 2007 [http://eclectic.ss.uci.edu/~drwhite/courses/Durbin-Watson.htm Using the Durbin–Watson test with multiple regression] ] for use with multiple regression provides a significance test for the presence of serial correlation in any series of consecutive observations, whether that of contiguity in the Standard Cross-Cultural Sample or in a time series.

Opportunities

In Anthropology, where Tylor's problem was first recognized by the statistican Galton in 1889, it is still not widely recognized that there are standard statistical adjustments for the problem of patches of similarity in observed cases and opportunities for new discoveries using autocorrelation methods. Some cross-cultural researchers (see, e.g., Korotayev and de Munck 2003) [Cite journal
url = http://www.anthrosource.net/doi/abs/10.1525/aa.2003.105.2.353?journalCode=aa
title = "Galton's Asset" and "Flower's Problem": Cultural Networks and Cultural Units in Cross-Cultural Research
author = Andrey Korotayev and Victor de Munck
journal = American Anthropologist
month = June
year = 2003
volume = 105
issue = 2
pages = 353–358
doi = 10.1525/aa.2003.105.2.353
format = abstract page] have begun to realize that evidence of diffusion, historical origin, and other sources of similarity among related societies or individuals should be renamed Galton’s Opportunity and Galton’s Asset rather than Galton’s Problem. Researchers, like Mace and Pagel (1994), now use longitudinal, cross-cultural, and regional variation analysis routinely to analyze all the competing hypotheses: functional relationships, diffusion, common historical origin, multilineal evolution, co-adaptation with environment, and complex social interaction dynamics.

Controversies

Within Anthropology, Galton's problem is often given as a cause to reject comparative studies altogether. Since the problem is a general one, common to the sciences and statistical inference generally, this particular criticism of cross-cultural or comparative studies – and there are many – is one that, logically speaking, amounts to a rejection of science and statistics altogether. Any data collected and analyzed by ethnographers, for example, is equally subject to Galton's problem, understood in its most general sense. A critique of the anticomparative critique is not limited to statistical comparison since it would apply as well to the analysis of text. That is, the analysis and use of text in argumentation is subject to critique as to the evidential basis of inference. Reliance purely on rhetoric is no protection against critique as to the validity of argument and its evidentiary basis.

There is little doubt, however, that the community of cross-cultural researchers have been remiss in ignoring Galton's problem. Expert investigation of this question shows results that "strongly suggest that the extensive reporting of naïve chi-square independence tests using cross-cultural data sets over the past several decades has led to incorrect rejection of null hypotheses at levels much higher than the expected 5% rate." [Cite journal
author = Malcolm Dow
year = 1993
title = Saving the Theory: Chi-squared Tests with Cross-Cultural Survey Data
journal = Cross-Cultural Research
volume = 27
issue = 3-4
pages = 247
doi = 10.1177/106939719302700305] The investigator concludes that "Incorrect theories that have been ‘saved’ by naïve chi-square tests with comparative data may yet be more rigorously tested another day.” [op. cit. p. 270.] Once again, the adjusted variance of a cluster sample is given as one multiplied by 1 + d(k+1) where k is the average size of a cluster, and a more complicated correction is given for the variance of contingency table correlations with r rows and c columns. Since this critique was published in 1993, and others like it, more authors have begun to adopt corrections for Galton's problem, but the majority in the cross-cultural field have not. Consequently, a large proportion of published results that rely on naive significance tests and that adopt the p <.05 rather than a p <.005 standard are likely to be in error because they are more susceptible to type II error, which is not to reject the null hypothesis when it is true.

Some cross-cultural researchers reject the seriousness of Galton's problem because, they argue, estimates of correlations and means may be unbiased even if autocorrelation, weak or strong, is present. Without investigating autocorrelation, however, they may still mis-estimate statistics dealing with relationships among variables. In regression analysis, for example, examining the patterns of autocorrelated residuals may give important clues to third factors that may affect the relationships among variables but that have not been included in the regression model. Second, if there areclusters of similar and related societies in the sample,measures of variance will be underestimated, leading tospurious statistical conclusions. for example, exaggeratingthe statistical significance of correlations. Third, theunderestimation of variance makes it difficult to test forreplication of results from two different samples, asthe results will more often be rejected as similar.

References

*Anselin, Luc. 1988. Spatial Econometrics: Methods and Models. Dordrecht: Kluwer Academic Publishers.
*Cliff, A.D., and J.K. Ord. 1973. Spatial Autocorrelation. London: Pion Press.
*Cliff, A.D. and J.K. Ord. 1981. Spatial Processes. London: Pion Press.
*Dow, Malcolm, Michael Burton, Douglas R. White and Karl Reitz. (1984). [http://www.anthrosource.net/doi/abs/10.1525/ae.1984.11.4.02a00080 Galton's problem as network autocorrelation] . American Ethnologist. 11:754-770.
*Interscience wiki. 2007. [http://intersci.ss.uci.edu/wiki/index.php/Comparative_research_tools#Using_Autocorrelation_in_model_specification Using Autocorrelation in model specification] (including software and tutorial)
*Jahn, Detlef. 2006. Globalization as "Galton's Problem": The Missing Link in the Analysis of the Diffusion Patterns in Welfare State Development. International Organization 60: (2): 401-431. [http://intersci.ss.uci.edu/wiki/index.php/Comparative_research pdf] [http://www.allacademic.com/meta/p69254_index.html abstract]
*Mace, Ruth & Pagel, Mark (1994), [http://www.journals.uchicago.edu/cgi-bin/resolve?doi=10.1086/204317&erFrom=8676762015401742140Guest The Comparative Method in Anthropology] . Current Anthropology 35(5):549-564.
*Stocking, George W. Jr. (1968). "Edward Burnett Tylor." International Encyclopedia of the Social Sciences. David L. Sills, editor, New York, Mcmillan Company: v.16, pp. 170-177.
*Student (W. S. Gosset). 1914. "The elimination of spurious correlation due to position in time or space". Biometrika 10:179-181.
*Witkowski, Stanley. 1974. "Galton's opportunity - hologeistic study of historical processes." Behavior Science Research 9(1):11-15.

Notes