- Impact evaluation
Impact evaluation assesses the changes in the well-being of individuals that can be attributed to a particular intervention, such as a project, program or policy. [ [http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/EXTISPMA/0,,menuPK:384336~pagePK:149018~piPK:149093~theSitePK:384329,00.html World Bank Poverty Group on Impact Evaluation] , accessed on January 6, 2008 ] This type of evaluation has received increasing attention in recent years. It is an important component of the armory of
evaluation tools and approaches, albeit only one among a number.Impact evaluation is more rigorous than many other types of evaluations, since it tries to establish a
counterfactual ("What would the situation have been if the intervention had not taken place?"). This involves the collection of baseline data for both an intervention group and a comparison group, as well as a second round of data collection after the intervention, which can be years later. Ideally the comparison group is selected randomly (in which case it is called acontrol group ), and it is isolated from any interventions similar to the original intervention, in order to generate an experimental design and to avoidselection bias . Random selection and isolation from interventions are seldom practicable and at least the latter is also ethically difficult to defend. Nevertheless, there exist impact evaluation designs without random selection that avoid the above-mentioned problems. Impact evaluation is particularly difficult for infrastructure projects, where random selection is more difficult and more time passes until impacts can be observed compared to interventions such as conditional cash transfers.Evaluation designs
Evaluation designs are determined by the choice of methods used to identify the counterfactual and can be broadly classified into three categories that vary in feasibility, cost, and degree of selection bias:
*Experimental design
*Quasi-experimental design
*Non-experimental designEstimation methods
Estimation methods broadly follow evaluation designs. Different designs require different estimation methods to measure the welfare of the counterfactual. The four categories of estimation methods are:
*Comparison of means
*Multi-variate regression
*Instrumental variables methods
*Double difference or difference-in-differenceOther definitions of impact evaluation
Impact evaluation is defined differently by different professionals, although some consensus has emerged around the definition quoted above.
Similar definitions
According to the US
Environmental Protection Agency impact evaluation is a form of evaluation that assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of a program. [ [http://www.epa.gov/evaluate/glossary/i-esd.htm US Environmental Protection Agency Program Evaluation Glossary] , accessed on January 6, 2008 ]According to the World Bank's Independent Evaluation Group (IEG), impact evaluation is the systematic identification of the effects positive or negative, intended or not on individual households, institutions, and the environment caused by a given development activity such as a program or project. [ [http://www.worldbank.org/ieg/ie/ World Bank Independent Evaluation Group] , accessed on January 6, 2008 ]
Different definitions
A less rigorous definition than the two previous definitions can be found on Evaluationwiki.org, where impact evaluation is simply defined as an evaluation that looks beyond the immediate results of policies, instruction, or services to identify longer-term as well as unintended program effects. [ [http://www.evaluationwiki.org/index.php/Evaluation_Definition:_What_is_Evaluation%3F#Impact_Evaluations Evaluation Definition: What is Evaluation? - EvaluationWiki ] ]
Examples of impact evaluations
The IEG of the World Bank has systematically assessed and summarized the experience of ten impact evaluation of development programs in various sectors carried out over the past 20 years. [ [http://lnweb18.worldbank.org/oed/oeddoclib.nsf/DocUNIDViewForJavaSearch/35BC420995BF58F8852571E00068C6BD/$file/impact_evaluation.pdf Impact Evaluation: The Experience of the Independent Evaluation Group of the World Bank, 2006] ]
An example of a well-known recent impact evaluation is the evaluation of the
Conditional Cash Transfer program PROGRESA (now calledOportunidades ) in Mexico on development outcomes, in particular schooling, immunization rates and child work.Sources and external links
* [http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/EXTISPMA/0,,menuPK:384336~pagePK:149018~piPK:149093~theSitePK:384329,00.html World Bank Poverty Group World Bank Poverty Group]
* [http://www.worldbank.org/ieg/ie/ World Bank Independent Evaluation Group] * Baker, Judy. 2000. [http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/EXTISPMA/0,,contentMDK:20194198~menuPK:451260~pagePK:148956~piPK:216618~theSitePK:384329,00.html Evaluating the Impact of Development Projects on Poverty: A Handbook for Practitioners.] Directions in Development, World Bank, Washington, D.C.See also
* Important publications in program evaluation
*Policy studies
*Policy analysis
*Participatory Impact Pathways Analysis
*Program evaluation
*Impact assessment References
Wikimedia Foundation. 2010.
