Paper presented at the Biennial Conference of the Society for Multivariate Analysis in the Behavioural Sciences, Jena, Germany
In item-centered methods for standard setting, judges indicate for each item the expected level of performance of a minimally competent student (MCS). The item judgements are combined into a cutoff on the test score scale. In examinee-centered methods, judges classify the examinees into different categories of mastery, such as masters versus nonmasters. A cutoff score is derived from a model that predicts category membership on the basis of the examinees' test performance in both groups.
In the present study, the standards derived from an item-centered and an examinee-centered method are compared on a test assessing the Flemish curriculum standards in biology for primary education. Using the revised Angoff method, primary school teachers indicated for each test item whether a MCS should be able to solve it correctly. Using the contrasting-group method, the same teachers also classified the pupils of their class into masters and nonmasters.
The two methods are compared using item response theory (IRT). Using the two-parameter IRT solution for the test, the judgements of the revised Angoff method can be transformed into an estimate of the location of the MCS on the ability scale (Van der Linden, 1982). A logistic regression of the estimated abilties of the students on their dichotomous classification by their teacher leads to a cutoff on the ability scale as well (Longford, 1996). A unified IRT model for the analysis of both methods of standard setting will be presented. The model is estimated in a Bayesian way using data augmented Gibbs sampling.