 Optimal design

This article is about the topic in the design of experiments. For the topic in optimal control theory, see shape optimization.
Optimal designs are a class of experimental designs that are optimal with respect to some statistical criterion.
In the design of experiments for estimating statistical models, optimal designs allow parameters to be estimated without bias and with minimumvariance. A nonoptimal design requires a greater number of experimental runs to estimate the parameters with the same precision as an optimal design. In practical terms, optimal experiments can reduce the costs of experimentation.
The optimality of a design depends on the statistical model and is assessed with respect to a statistical criterion, which is related to the variancematrix of the estimator. Specifying an appropriate model and specifying a suitable criterion function both require understanding of statistical theory and practical knowledge with designing experiments.
Optimal designs are also called optimum designs.^{[2]}
Contents
Advantages of optimal designs
Optimal designs offer three advantages over suboptimal experimental designs:^{[3]}
 Optimal designs reduce the costs of experimentation by allowing statistical models to be estimated with fewer experimental runs.
 Optimal designs can accommodate multiple types of factors, such as process, mixture, and discrete factors.
 Designs can be optimized when the designspace is constrained, for example, when the mathematical processspace contains factorsettings that are practically infeasible (e.g. due to safety concerns).
Risks of optimal designs
 Users of optimal designs should use caution because unlike factorial and orthogonal designs, main effects are not always estimable. Thus, there is a risk of confusing the main effects of specific parameters. ^{[4]}
Minimizing the variance of estimators
Experimental designs are evaluated using statistical criteria.^{[5]}
It is known that the least squares estimator minimizes the variance of meanunbiased estimators (under the conditions of the Gauss–Markov theorem). In the estimation theory for statistical models with one real parameter, the reciprocal of the variance of an ("efficient") estimator is called the "Fisher information" for that estimator.^{[6]} Because of this reciprocity, minimizing the variance corresponds to maximizing the information.
When the statistical model has several parameters, however, the mean of the parameterestimator is a vector and its variance is a matrix. The inverse matrix of the variancematrix is called the "information matrix". Because the variance of the estimator of a parameter vector is a matrix, the problem of "minimizing the variance" is complicated. Using statistical theory, statisticians compress the informationmatrix using realvalued summary statistics; being realvalued functions, these "information criteria" can be maximized.^{[7]} The traditional optimalitycriteria are invariants of the information matrix; algebraically, the traditional optimalitycriteria are functionals of the eigenvalues of the information matrix.
 Aoptimality ("average" or trace)
 Coptimality
 Doptimality (determinant)
 A popular criterion is Doptimality, which seeks to minimize (X'X)^{−1}, or equivalently maximize the determinant of the information matrix X'X of the design. This criterion results in maximizing the differential Shannon information content of the parameter estimates.
 Eoptimality (eigenvalue)
 Another design is Eoptimality, which maximizes the minimum eigenvalue of the information matrix. The Eoptimality criterion need not be differentiable at every point. Such Eoptimal designs can be computed using methods of convex minimization that use subgradients rather than gradients at points of nondifferentiability. Any nondifferentiability need not be a serious problem, however: Eoptimality problems are special cases of semidefiniteprogramming problems which have effective solutionmethods, especially bundle methods and interiorpoint methods.
 Toptimality
 This criterion maximizes the trace of the information matrix.
Other optimalitycriteria are concerned with the variance of predictions:
 Goptimality
 A popular criterion is Goptimality, which seeks to minimize the maximum entry in the diagonal of the hat matrix X(X'X)^{−1}X'. This has the effect of minimizing the maximum variance of the predicted values.
 Ioptimality (integrated)
 A second criterion on prediction variance is Ioptimality, which seeks to minimize the average prediction variance over the design space.
 Voptimality (variance)
 A third criterion on prediction variance is Voptimality, which seeks to minimizes the average prediction variance over a set of m specific points.^{[8]}
Contrasts
Main article: Contrast (statistics)See also: Nuisance parameterIn many applications, the statistician is most concerned with a "parameter of interest" rather than with "nuisance parameters". More generally, statisticians consider linear combinations of parameters, which are estimated via linear combinations of treatmentmeans in the design of experiments and in the analysis of variance; such linear combinations are called contrasts. Statisticians can use appropriate optimalitycriteria for such parameters of interest and for more generally for contrasts.^{[9]}
Finding optimal designs
Catalogs of optimal designs occur in books and in software libraries.
In addition, major statistical systems like SAS and R have procedures for optimizing a design according to a user's specification. The experimenter must specify a model for the design and an optimalitycriterion before the method can compute an optimal design.^{[10]}
Practical considerations
Some advanced topics in optimal design require more statistical theory and practical knowledge in designing experiments.
Model dependence and robustness
Since the optimality criterion of most optimal designs is based on some function of the information matrix, the 'optimality' of a given design is model dependent: While an optimal design is best for that model, its performance may deteriorate on other models. On other models, an optimal design can be either better or worse than a nonoptimal design.^{[11]} Therefore, it is important to benchmark the performance of designs under alternative models.^{[12]}
Choosing an optimality criterion and robustness
The choice of an appropriate optimality criterion requires some thought, and it is useful to benchmark the performance of designs with respect to several optimality criteria. Cornell writes that
since the [traditional optimality] criteria . . . are varianceminimizing criteria, . . . a design that is optimal for a given model using one of the . . . criteria is usually nearoptimal for the same model with respect to the other criteria.^{[13]}
Indeed, there are several classes of designs for which all the traditional optimalitycriteria agree, according to the theory of "universal optimality" of Kiefer.^{[14]} The experience of practitioners like Cornell and the "universal optimality" theory of Kiefer suggest that robustness with respect to changes in the optimalitycriterion is much greater than is robustness with respect to changes in the model.
Flexible optimality criteria and convex analysis
Highquality statistical software provide a combination of libraries of optimal designs or iterative methods for constructing approximately optimal designs, depending on the model specified and the optimality criterion. Users may use a standard optimalitycriterion or may program a custommade criterion.
All of the traditional optimalitycriteria are convex (or concave) functions, and therefore optimaldesigns are amenable to the mathematical theory of convex analysis and their computation can use specialized methods of convex minimization.^{[15]} The practitioner need not select exactly one traditional, optimalitycriterion, but can specify a custom criterion. In particular, the practitioner can specify a convex criterion using the maxima of convex optimalitycriteria and nonnegative combinations of optimality criteria (since these operations preserve convex functions). For convex optimality criteria, the KieferWolfowitz equivalence theorem allows the practitioner to verify that a given design is globally optimal.^{[16]} The KieferWolfowitz equivalence theorem is related with the LegendreFenchel conjugacy for convex functions.^{[17]}
If an optimalitycriterion lacks convexity, then finding a global optimum and verifying its optimality often are difficult.
Model uncertainty and Bayesian approaches
Model selection
See also: model selectionWhen scientists wish to test several theories, then a statistician can design an experiment that allows optimal tests between specified models. Such "discrimination experiments" are especially important in the biostatistics supporting pharmacokinetics and pharmacodynamics, following the work of Cox and Atkinson.^{[18]}
Bayesian experimental design
Main article: Bayesian experimental designWhen practitioners need to consider multiple models, they can specify a probabilitymeasure on the models and then select any design maximizing the expected value of such an experiment. Such probabilitybased optimaldesigns are called optimal Bayesian designs. Such Bayesian designs are used especially for generalized linear models (where the response follows an exponentialfamily distribution).^{[19]}
The use of a Bayesian design does not force statisticians to use Bayesian methods to analyze the data, however. Indeed, the "Bayesian" label for probabilitybased experimentaldesigns is disliked by some researchers.^{[20]} Alternative terminology for "Bayesian" optimality includes "onaverage" optimality or "population" optimality.
Iterative experimentation
Scientific experimentation is an iterative process, and statisticians have developed several approaches to the optimal design of sequential experiments.
Sequential analysis
Main article: Sequential analysisSequential analysis was pioneered by Abraham Wald.^{[21]} In 1972, Herman Chernoff wrote an overview of optimal sequential designs,^{[22]} while adaptive designs were surveyed later by S. Zacks.^{[23]} Of course, much work on the optimal design of experiments is related to the theory of optimal decisions, especially the statistical decision theory of Abraham Wald.^{[24]}
Responsesurface methodology
Main article: Response surface methodologyOptimal designs for responsesurface models are discussed in the textbook by Atkinson, Donev and Tobias, and in the survey of Gaffke and Heiligers and in the mathematical text of Pukelsheim. The blocking of optimal designs is discussed in the textbook of Atkinson, Donev and Tobias and also in the monograph by Goos.
The earliest optimal designs were developed to estimate the parameters of regression models with continuous variables, for example, by J. D. Gergonne in 1815 (Stigler). In English, two early contributions were made by Charles S. Peirce and Kirstine Smith.
Pioneering designs for multivariate responsesurfaces were proposed by George E. P. Box. However, Box's designs have few optimality properties. Indeed, the BoxBehnken design requires excessive experimental runs when the number of variables exceeds three.^{[25]} Box's "centralcomposite" designs require more experimental runs than do the optimal designs of Kôno.^{[26]}
System identification and stochastic approximation
See also: System identification and Stochastic approximationThe optimization of sequential experimentation is studied also in stochastic programming and in systems and control. Popular methods include stochastic approximation and other methods of stochastic optimization. Much of this research has been associated with the subdiscipline of system identification.^{[27]} In computational optimal control, D. Judin & A. Nemirovskii and Boris Polyak has described methods that are more efficient than the (Armijostyle) stepsize rules introduced by G. E. P. Box in responsesurface methodology^{[28]}
Adaptive designs are used in clinical trials, and optimal adaptive designs are surveyed in the Handbook of Experimental Designs chapter by Shelemyahu Zacks.
Specifying the number of experimental runs
Using a computer to find a good design
There are several methods of finding an optimal design, given an a priori restriction on the number of experimental runs or replications. Some of these methods are discussed by Atkinson, Donev and Tobias and in the paper by Hardin and Sloane. Of course, fixing the number of experimental runs a priori would be impractical. Prudent statisticians examine the other optimal designs, whose number of experimental runs differ.
Discretizing probabilitymeasure designs
In the mathematical theory on optimal experiments, an optimal design can be a probability measure that is supported on an infinite set of observationlocations. Such optimal probabilitymeasure designs solve a mathematical problem that neglected to specify the cost of observations and experimental runs. Nonetheless, such optimal probabilitymeasure designs can be discretized to furnish approximately optimal designs.^{[29]}
In some cases, a finite set of observationlocations suffices to support an optimal design. Such a result was proved by Kôno and Kiefer in their works on responsesurface designs for quadratic models. The KônoKiefer analysis explains why optimal designs for responsesurfaces can have discrete supports, which are very similar as do the less efficient designs that have been traditional in response surface methodology.^{[30]}
History
The prophet of scientific experimentation, Francis Bacon, foresaw that experimental designs should be improved. Researchers who improved experiments were praised in Bacon's utopian novel New Atlantis:
Then after divers meetings and consults of our whole number, to consider of the former labors and collections, we have three that take care out of them to direct new experiments, of a higher light, more penetrating into nature than the former. These we call lamps.
In 1815, an article on optimal designs for polynomial regression was published by Joseph Diaz Gergonne, according to Stigler.
Charles S. Peirce proposed an economic theory of scientific experimentation in 1876, which sought to maximize the precision of the estimates. Peirce's optimal allocation immediately improved the accuracy of gravitational experiments and was used for decades by Peirce and his colleagues. In his 1882 published lecture at Johns Hopkins University, Peirce introduced experimental design with these words:
Logic will not undertake to inform you what kind of experiments you ought to make in order best to determine the acceleration of gravity, or the value of the Ohm; but it will tell you how to proceed to form a plan of experimentation.
[....] Unfortunately practice generally precedes theory, and it is the usual fate of mankind to get things done in some boggling way first, and find out afterward how they could have been done much more easily and perfectly.^{[31]}Like Bacon, Peirce was aware that experimental methods should strive for substantial improvement (even optimality).
Kirstine Smith proposed optimal designs for polynomial models in 1918. (Kirstine Smith had been a student of the Danish statistician Thorvald N. Thiele and was working with Karl Pearson in London.)
See also
 Efficiency (statistics)
 Entropy (information theory)
 Fisher information
 Glossary of experimental design
 Hadamard's maximal determinant problem
 Information theory
Notes
 ^ Nordström (1999, p. 176)
 ^ The adjective "optimum" (and not "optimal") "is the slightly older form in English and avoids the construction 'optim(um) + al´there is no 'optimalis' in Latin" (page x in Optimum Experimental Designs, with SAS, by Atkinson, Donev, and Tobias).
 ^ These three advantages (of optimal designs) are documented in the textbook by Atkinson, Donev, and Tobias.
 ^ This risk of optimal designs is described by NIST http://www.itl.nist.gov/div898/handbook/pri/section5/pri521.htm as well as by numerous statistical articles
 ^ Such criteria are called objective functions in optimization theory.
 ^ The Fisher information and other "information" functionals are fundamental concepts in statistical theory.
 ^ Traditionally, statistians have evaluated estimators and designs by considering some summary statistic of the covariance matrix (of a meanunbiased estimator), usually with positive real values (like the determinant or matrix trace). Working with positive realnumbers brings several advantages: If the estimator of a single parameter has a positive variance, then the variance and the Fisher information are both positive real numbers; hence they are members of the convex cone of nonnegative real numbers (whose nonzero members have reciprocals in this same cone).
For several parameters, the covariancematrices and informationmatrices are elements of the convex cone of nonnegativedefinite symmetric matrices in a partially ordered vector space, under the Loewner (Löwner) order. This cone is closed under matrixmatrix addition, under matrixinversion, and under the multiplication of positive realnumbers and matrices. An exposition of matrix theory and the Loewnerorder appears in Pukelsheim.  ^ The above optimalitycriteria are convex functions on domains of symmetric positivesemidefinite matrices: See an online textbook for practitioners, which has many illustrations and statistical applications:
 Boyd, Stephen P.; Vandenberghe, Lieven (2004) (pdf). Convex Optimization. Cambridge University Press. ISBN 9780521833783. http://www.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf. Retrieved October 15, 2011. (book in pdf)
 ^ Optimality criteria for "parameters of interest" and for contrasts are discussed by Atkinson, Donev and Tobias.
 ^ Iterative methods and approximation algorithms are surveyed in the textbook by Atkinson, Donev and Tobias and in the monographs of Fedorov (historical) and Pukelsheim, and in the survey article by Gaffke and Heiligers.
 ^ See Kiefer ("Optimum Designs for Fitting Biased Multiresponse Surfaces" pages 289299).
 ^ Such benchmarking is discussed in the textbook by Atkinson et al.and in the papers of Kiefer. Modelrobust designs (including "Bayesian" designs) are surveyed by Chang and Notz.
 ^
 Cornell, John (2002). Experiments with Mixtures: Designs, Models, and the Analysis of Mixture Data (third ed.). Wiley. ISBN 0471079162. (Pages 400401)
 ^ An introduction to "universal optimality" appears in the textbook of Atkinson, Donev, and Tobias. More detailed expositions occur in the advanced textbook of Pukelsheim and the papers of Kiefer.
 ^ Computational methods are discussed by Pukelsheim and by Gaffke and Heiligers.
 ^ The KieferWolfowitz equivalence theorem is discussed in Chapter 9 of Atkinson, Donev, and Tobias.
 ^ Pukelsheim uses convex analysis to study KieferWolfowitz equivalence theorem in relation to the LegendreFenchel conjugacy for convex functions The minimization of convex functions on domains of symmetric positivesemidefinite matrices is explained in an online textbook for practitioners, which has many illustrations and statistical applications:
 Convex Optimization. Cambridge University Press. 2004. http://www.stanford.edu/~boyd/cvxbook/. (book in pdf)
 ^ See Chapter 20 in Atkinison, Donev, and Tobias.
 ^ Bayesian designs are discussed in Chapter 18 of the textbook by Atkinson, Donev, and Tobias. More advanced discussions occur in the monograph by Fedorov and Hackl, and the articles by Chaloner and Verdinelli and by DasGupta. Bayesian designs and other aspects of "modelrobust" designs are discussed by Chang and Notz.
 ^ As an alternative to "Bayesian optimality", "onaverage optimality" is advocated in Fedorov and Hackl.
 ^
 Wald, Abraham (June 1945). "Sequential Tests of Statistical Hypotheses". The Annals of Mathematical Statistics 16 (2): 117–186. doi:10.1214/aoms/1177731118. JSTOR 2235829.
 ^ Chernoff, H. (1972) Sequential Analysis and Optimal Design, SIAM Monograph.
 ^ Zacks, S. (1996) "Adaptive Designs for Parametric Models". In: Ghosh, S. and Rao, C. R., (Eds) (1996). Design and Analysis of Experiments, Handbook of Statistics, Volume 13. NorthHolland. ISBN 0444820612. (pages 151–180)
 ^ Henry P. Wynn wrote, "the modern theory of optimum design has its roots in the decision theory school of U.S. statistics founded by Abraham Wald" in his introduction "Jack Kiefer's Contributions to Experimental Design", which is pages xviixxiv in the following volume:
 Kiefer, Jack Carl. (1985). Brown, Lawrence D. and Olkin, Ingram and Jerome Sacks and Wynn, Henry P. ed. Jack Carl Kiefer Collected Papers III Design of Experiments. SpringerVerlag and the Institute of Mathematical Statistics. pp. 718+xxv. ISBN 038796004X.
 Kiefer, J. (1959). "Optimum Experimental Designs". Journal of the Royal Statistical Society, Series B (Methodology) 21: 272–319.
 ^ In the field of response surface methodology, the inefficiency of the BoxBehnken design is noted by Wu and Hamada (page 422).
 Wu, C. F. Jeff and Hamada, Michael (2002). Experiments: Planning, Analysis, and Parameter Design Optimization. Wiley. ISBN 0471255114.
 ^ The inefficiency of Box's "centralcomposite" designs are discussed by according to Atkinson, Donev, and Tobias (page 165). These authors also discuss the blocking of Kônotype designs for quadratic responsesurfaces.
 ^ In system identification, the following books have chapters on optimal experimental design:
 Goodwin, Graham C. and Payne, Robert L. (1977). Dynamic System Identification: Experiment Design and Data Analysis. Academic Press. ISBN 0122897501.
 Walter, Éric and Pronzato, Luc (1997). Identification of Parametric Models from Experimental Data. Springer.
 ^ Some stepsize rules for of Judin & Nemirovskii and of Polyak are explained in the textbook by Kushner and Yin:
 Kushner, Harold J. and Yin, G. George (2003). Stochastic Approximation and Recursive Algorithms and Applications (Second ed.). Springer. ISBN 0387008942.
 ^ The discretization of optimal probabilitymeasure designs to provide approximately optimal designs is discussed by Atkinson, Donev, and Tobias and by Pukelsheim (especially Chapter 12).
 ^ Regarding designs for quadratic responsesurfaces, the results of Kôno and Kiefer are discussed in Atkinson, Donev, and Tobias. Mathematically, such results are associated with Chebyshev polynomials, "Markov systems", and "moment spaces": See
 Karlin, Samuel and Shapley, Lloyd (1953). "Geometry of moment spaces". Mem. Amer. Math. Soc. 12.
 Karlin, Samuel and Studden, William J. (1966). Tchebycheff systems: With applications in analysis and statistics. WileyInterscience.
 Dette, Holger and Studden, William J. (1997). The Theory of canonical moments with applications in statistics, probability, and analysis. John Wiley & Sons Inc..
 ^ Peirce, C. S. (1882), "Introductory Lecture on the Study of Logic" delivered September 1882, published in Johns Hopkins University Circulars, v. 2, n. 19, pp. 11–12, November 1882, see p. 11, Google Books Eprint. Reprinted in Collected Papers v. 7, paragraphs 59–76, see 59, 63, Writings of Charles S. Peirce v. 4, pp. 378–82, see 378, 379, and The Essential Peirce v. 1, pp. 210–14, see 21011, also lower down on 211.
References
 Atkinson, A. C.; Donev, A. N.; Tobias, R. D. (2007). Optimum experimental designs, with SAS. Oxford University Press. pp. 511+xvi. ISBN 9780199296606. http://books.google.se/books?id=oIHsrw6NBmoC.
 Chernoff, Herman (1972). Sequential analysis and optimal design. Society for Industrial and Applied Mathematics. ISBN 0898710065.
 Fedorov, V. V. (1972). Theory of Optimal Experiments. Academic Press.
 Fedorov, Valerii V.; Hackl, Peter (1997). ModelOriented Design of Experiments. Lecture Notes in Statistics. 125. SpringerVerlag.
 Goos, Peter (2002). The Optimal Design of Blocked and Splitplot Experiments. Lecture Notes in Statistics. 164. Springer. http://users.telenet.be/peter.goos/springer.htm.
 Kiefer, Jack Carl (1985). Brown; Olkin, Ingram; Sacks, Jerome et al.. eds. Jack Carl Kiefer: Collected papers III—Design of experiments. SpringerVerlag and the Institute of Mathematical Statistics. pp. 718+xxv. ISBN 038796004X.
 Logothetis, N.; Wynn, H. P. (1989). Quality through design: Experimental design, offline quality control, and Taguchi's contributions. Oxford U. P.. pp. 464+xi. ISBN 0198519931.
 Nordström, Kenneth (May 1999). "The life and work of Gustav Elfving". Statistical Science 14 (2): 174–196. doi:10.1214/ss/1009212244. JSTOR 2676737. MR1722074. http://projecteuclid.org/euclid.ss/1009212244%7C.
 Pukelsheim, Friedrich (2006). Optimal design of experiments. Classics in Applied Mathematics. 50 (republication with erratalist and new preface of Wiley (047161971X) 1993 ed.). SIAM (Society for Industrial and Applied Mathematics). pp. 454+xxxii. ISBN 9780898716047. http://books.google.se/books?id=5ZcfDZUJ4F8C.
 Shah, Kirti R. and Sinha, Bikas K. (1989). Theory of Optimal Designs. Lecture Notes in Statistics. 54. SpringerVerlag. pp. 171+viii. ISBN 0387969918.
Further reading
Textbooks for practitioners and students
Textbooks emphasizing regression and responsesurface methodology
The textbook by Atkinson, Donev and Tobias has been used for short courses for industrial practitioners as well as university courses.
 Atkinson, A. C.; Donev, A. N.; Tobias, R. D. (2007). Optimum experimental designs, with SAS. Oxford University Press. pp. 511+xvi. ISBN 9780199296606. http://books.google.se/books?id=oIHsrw6NBmoC.
 Logothetis, N.; Wynn, H. P. (1989). Quality through design: Experimental design, offline quality control, and Taguchi's contributions. Oxford U. P.. pp. 464+xi. ISBN 0198519931.
Textbooks emphasizing block designs
Optimal block designs are discussed by Bailey and by Bapat. The first chapter of Bapat's book reviews the linear algebra used by Bailey (or the advanced books below). Bailey's exercises and discussion of randomization both emphasize statistical concepts (rather than algebraic computations).
 Bailey, R. A. (2008). Design of Comparative Experiments. Cambridge U. P.. ISBN 9780521683579. http://www.maths.qmul.ac.uk/~rab/DOEbook. Draft available online. (Especially Chapter 11.8 "Optimality")
 Bapat, R. B. (2000). Linear Algebra and Linear Models (Second ed.). Springer. ISBN 9780387988719. http://books.google.se/books?id=T5dsExIP3aAC. (Chapter 5 "Block designs and optimality", pages 99–111)
Optimal block designs are discussed in the advanced monograph by Shah and Sinha and in the surveyarticles by Cheng and by Majumdar, which are cited below.
Books for professional statisticians and researchers
 Chernoff, Herman (1972). Sequential Analysis and Optimal Design. SIAM. ISBN 0898710065.
 Fedorov, V. V. (1972). Theory of Optimal Experiments. Academic Press.
 Fedorov, Valerii V. and Hackl, Peter (1997). ModelOriented Design of Experiments. Lecture Notes in Statistics. 125. SpringerVerlag.
 Goos, Peter (2002). The Optimal Design of Blocked and Splitplot Experiments. Lecture Notes in Statistics. 164. Springer. http://users.telenet.be/peter.goos/springer.htm.
 Kiefer, Jack Carl. (1985). Brown, Lawrence D. and Olkin, Ingram and Jerome Sacks and Wynn, Henry P. ed. Jack Carl Kiefer Collected Papers III Design of Experiments. SpringerVerlag and the Institute of Mathematical Statistics. pp. 718+xxv. ISBN 038796004X.
 Pukelsheim, Friedrich (2006). Optimal Design of Experiments. Classics in Applied Mathematics. 50 (republication with erratalist and new preface of Wiley (047161971X) 1993 ed.). SIAM(Society for Industrial and Applied Mathematics). pp. 454+xxxii. ISBN 9780898716047. http://books.google.se/books?id=5ZcfDZUJ4F8C.
 Shah, Kirti R. and Sinha, Bikas K. (1989). Theory of Optimal Designs. Lecture Notes in Statistics. 54. SpringerVerlag. pp. 171+viii. ISBN 0387969918.
Articles and chapters
 Chaloner, Kathryn and Verdinelli, Isabella (1995). "Bayesian Experimental Design: A Review". Statistical Science 10 (3): 273–304. doi:10.1214/ss/1177009939.
 Ghosh, S. and Rao, C. R., ed (1996). Design and Analysis of Experiments. Handbook of Statistics. 13. NorthHolland. ISBN 0444820612.
 Chang, C.S. and Notz, W. I. "Model Robust Designs". pp. 1055–1099.
 Cheng, C.S. "Optimal Design: Exact Theory". pp. 977–1006.
 DasGupta, A. "Review of Optimal Bayesian Designs". pp. 1099–1148.
 Gaffke, N. and Heiligers, B. "Approximate Designs for Polynomial Regression: Invariance, Admissibility, and Optimality". pp. 1149–1199.
 Majumdar, D. "Optimal and Efficient TreatmentControl Designs". pp. 1007–1054.
 Stufken, J. "Optimal Crossover Designs". pp. 63–90.
 Zacks, S. "Adaptive Designs for Parametric Models". pp. 151–180.
 R. H. Hardin and N. J. A. Sloane, "A New Approach to the Construction of Optimal Designs", Journal of Statistical Planning and Inference, vol. 37, 1993, pp. 339369
 Kôno, Kazumasa (1962). "Optimum designs for quadratic regression on kcube". Memoirs of the Faculty of Science. Kyushu University. Series A. Mathematics 16 (2): 114–122. doi:10.2206/kyushumfs.16.114. http://www.ams.org/mathscinet/pdf/153090.pdf.
Historical
 Gergonne, J. D. (November 1974 [1815]). "The application of the method of least squares to the interpolation of sequences". Historia Mathematica 1 (4): 439–447. doi:10.1016/03150860(74)900342. http://www.sciencedirect.com/science/article/B6WG94D7JMHH20/2/df451ec5fbb7c044d0f4d900af80ec86.
 Stigler, Stephen M. (November 1974). "Gergonne's 1815 paper on the design and analysis of polynomial regression experiments". Historia Mathematica 1 (4): 431–439. doi:10.1016/03150860(74)900330. http://www.sciencedirect.com/science/article/B6WG94D7JMHH1Y/2/680c7ada0198761e9866197d53512ab4.
 Peirce, C. S (1876). "Note on the Theory of the Economy of Research". Coast Survey Report: 197–201. (Appendix No. 14). NOAA PDF Eprint. Reprinted in Collected Papers of Charles Sanders Peirce. 7. 1958. paragraphs 139–157, and in Peirce, C. S. (July–August 1967). "Note on the Theory of the Economy of Research". Operations Research 15 (4): 643–648. doi:10.1287/opre.15.4.643. Abstract at JSTOR.
 Smith, Kirstine (1918). "On the Standard Deviations of Adjusted and Interpolated Values of an Observed Polynomial Function and its Constants and the Guidance They Give Towards a Proper Choice of the Distribution of the Observations". Biometrika 12 (1/2): 1–85. JSTOR 2331929.
Statistics Descriptive statistics Summary tablesPearson productmoment correlation · Rank correlation (Spearman's rho, Kendall's tau) · Partial correlation · Scatter plotBar chart · Biplot · Box plot · Control chart · Correlogram · Forest plot · Histogram · QQ plot · Run chart · Scatter plot · Stemplot · Radar chartData collection Designing studiesDesign of experiments · Factorial experiment · Randomized experiment · Random assignment · Replication · Blocking · Optimal designUncontrolled studiesStatistical inference Frequentist inferenceSpecific testsZtest (normal) · Student's ttest · Ftest · Pearson's chisquared test · Wald test · Mann–Whitney U · Shapiro–Wilk · Signedrank · Kolmogorov–Smirnov testCorrelation and regression analysis Errors and residuals · Regression model validation · Mixed effects models · Simultaneous equations modelsNonstandard predictorsPartition of varianceCategorical, multivariate, timeseries, or survival analysis Decomposition (Trend · Stationary process) · ARMA model · ARIMA model · Vector autoregression · Spectral density estimationApplications Categories: Design of experiments
 Statistical methods
 Regression analysis
 Statistical tests
 Statistical theory
 Optimal decisions
 Mathematical optimization
 Operations research
 Industrial engineering
 Systems engineering
 Quality control
 Quality
Wikimedia Foundation. 2010.