Standard-setting study

Standard-setting study

A "standard-setting study" is an official research study conducted by an organization that sponsors tests to determine a cutscore for the test. To be legally defensible and meet the Standards for Educational and Psychological Testing, a cutscore cannot be arbitrarily determined, it must be empirically justified. For example, the organization cannot merely decide that the cutscore will be 70% correct. Instead, a study is conducted to determine what score best differentiates the classifications of examinees, such as competent vs. incompetent.

Standard-setting studies are often performed using focus groups of 5-15 subject matter experts that represent the testing organization.

Types of standard-setting studies

Standard-setting studies fall into two categories, item-centered and person-centered. Examples of item-centered methods include the Angoff, Ebel, Nedelsky, and Bookmark methods, while examples of person-centered methods include the Borderline Survey and Contrasting Groups approaches. These are so categorized by the focus of the analysis; in item-centered studies, the organization evaluates items with respect to a given population of persons, and vice versa for person-centered studies.

Item-centered studies

The Angoff approach is very widely used. [Zieky, M.J. (2001). So much has changed: how the setting of cutscores has evolved since the 1980s. In Cizek, G.J. (Ed.), "Setting Performance Standards", p. 19-52. Mahwah, NJ: Lawrence Erlbaum Associates.] This method requires the assembly of a group of subject matter experts, who are asked to evaluate each item and estimate the proportion of minimally competent examinees that would correctly answer the item. The ratings are averaged across raters for each item and then summed to obtain a panel-recommended raw cutscore. This cutscore then represents the score which the panel estimates a minimally competent candidate would get.

The Bookmark method is another widely used item-centered approach. Items in a test (or a subset of them) are ordered by difficulty, and each expert places a "bookmark" in the sequence at the location of the cutscore. [Lewis, D. M., Mitzel, H. C., Green, D. R. (June, 1996). Standard Setting: A Bookmark Approach. In D. R. Green (Chair), IRT-Based Standard-Setting Procedures Utilizing Behavioral Anchoring. Paper presented at the 1996 Council of Chief State School Officers National Conference on Large Scale Assessment, Phoenix, AZ.] [Mitzel, H. C., Lewis, D. M., Patz, R. J., & Green, D. R. (2000). The Bookmark Procedure: Cognitive Perspectives on Standard Setting. Chapter in Setting Performance Standards: Concepts, Methods, and Perspectives (G. J. Cizek, ed.). Mahwah, NJ: Lawrence Erlbaum Associates.]

Person-centered studies

Rather than the items that distinguish competent candidates, person-centered studies evaluate the examinees themselves. While this might seem more appropriate, it is often more difficult because examinees are not a captive population, as is a list of items.

For example, a if a new test comes out regarding new content (as often happens in information technology tests), the test could be given to an initial sample called a beta sample, along with a survey of professional characteristics. The testing organization could then analyze and evaluate the relationship between the test scores and important statistics, such as skills, education, and experience. The cutscore could be set as the score that best differentiates between those examinees characterized as "passing" and those as "failing."

References


Wikimedia Foundation. 2010.

Игры ⚽ Поможем написать реферат

Look at other dictionaries:

  • Orthodox Study Bible — Full name: Orthodox Study Bible Abbreviation: OSB Complete Bible published: 2008 Textual basis: NT: the NKJV, from the Textus Receptus, a part of the Majority Text family of 94% of all Greek manuscripts. High Correspondence to the Stephanus 1550… …   Wikipedia

  • Observational study — In epidemiology and statistics, an observational study draws inferences about the possible effect of a treatment on subjects, where the assignment of subjects into a treated group versus a control group is outside the control of the… …   Wikipedia

  • time and motion study — noun an analysis of a specific job in an effort to find the most efficient method in terms of time and effort • Syn: ↑time and motion study, ↑time motion study, ↑motion study, ↑time study, ↑work study • Hypernyms: ↑examination, ↑ …   Useful english dictionary

  • New Source Performance Standard — New Source Performance Standards (NSPS) are pollution control standards issued by the United States Environmental Protection Agency (EPA). The term is used in the Clean Air Act Extension of 1970 (CAA) to refer to air pollution emission standards …   Wikipedia

  • Institute for the Study of Labor — The Institute for the Study of Labor is a private, independent economic research institute. It was founded under the legal form of a limited liability company. Its German name is Forschungsinstitut zur Zukunft der Arbeit hence the abbreviation… …   Wikipedia

  • Cutscore — A cutscore, also known as a passing score or passing point, is a single point on a score continuum that differentiates between classifications along the continuum. The most common cutscore, that many are familiar with, is a score that… …   Wikipedia

  • Standards for Educational and Psychological Testing — The Standards for Educational and Psychological Testing is a set of testing standards developed jointly by the American Educational Research Association (AERA), American Psychological Association (APA), and the National Council on Measurement in… …   Wikipedia

  • performing arts — arts or skills that require public performance, as acting, singing, or dancing. [1945 50] * * * ▪ 2009 Introduction Music Classical.       The last vestiges of the Cold War seemed to thaw for a moment on Feb. 26, 2008, when the unfamiliar strains …   Universalium

  • Western architecture — Introduction       history of Western architecture from prehistoric Mediterranean cultures to the present.       The history of Western architecture is marked by a series of new solutions to structural problems. During the period from the… …   Universalium

  • human rights — fundamental rights, esp. those believed to belong to an individual and in whose exercise a government may not interfere, as the rights to speak, associate, work, etc. [1785 95] * * * Rights that belong to an individual as a consequence of being… …   Universalium

Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”