Title: Real-time feedback on rater drift in constructed-response items: An example from the Golden State Examination
Authors: Hoskens, Machteld ×
Wilson, M #
Issue Date: 2001
Publisher: Natl counc meas educ
Series Title: Journal of educational measurement vol:38 issue:2 pages:121-145
Abstract: In this study, patterns of variation in severities of a group of raters over time or so-called "rater drift" was examined when raters scored an essay written under examination conditions. At the same time feedback was given to rater leaders (called "table leaders") who then interpreted the feedback and reported to the raters. Rater severities in five successive periods were estimated using a modified linear logistic test model (LLTM, Fischer 1973) approach. It was found that the raters did indeed drift towards the mean, but a planned comparision of the feedback with a control condition was not successful; it was believed that this was due to contamination at the table leader level. A series of models was also estimated designed to detect other types of rater effects beyond severity: a tendency to use extreme scores, and tendency to prefer certain categories. The models for these effects were found to be showing significant improvement in fit, implying that these effects were indeed present, although they were difficult to detect in relatively short time periods.
ISSN: 0022-0655
Publication status: published
KU Leuven publication type: IT
Appears in Collections:Department of Managerial Economics, Strategy and Innovation (MSI), Leuven
× corresponding author
# (joint) last author

Files in This Item:

There are no files associated with this item.

Request a copy


All items in Lirias are protected by copyright, with all rights reserved.

© Web of science