European Conference of Methodology edition:5 location:Santiago de Compostela date:17-20 July 2012
The present study analyzes item position effects within the PISA 2009 reading assessment. In PISA, items are presented in a balanced booklet design with each booklet consisting of four blocks of items with each item being presented equally often at all of the four possible block positions within the booklet. Thus, item position and item content are varied independently, allowing to analyze the effect of item position on item difficulty. The item responses were analyzed using a logistic multilevel model treating responses as level one variables nested within students, and students nested within schools. With only fixed effects for item content, this model is equivalent to a multilevel Rasch model. In addition to the item difficulties the model was extended to include the main effect of the item position within the booklet and an associated random effect to capture individual differences in item position effects, i.e. the students’ persistence during the assessment. The model allows an estimation of (1) the general effect of the item position, (2) the amount of variance in persistence within and between schools, and (3) the correlation between the performance level and the persistence within and between schools. The analyses make use of the published reading assessment data from the PISA 2009 study consisting of data from N = 475.460 students from 65 countries. Responses to 95 reading test items were used in the analysis. All effects of interest were analyzed separately for each country, allowing to compare the effects across countries. A significant effect of item position on item difficulties was found consistently across all analyzed countries, however the effect is more pronounced for countries with a lower international performance level. The individual differences in persistence are relatively small in all countries, but more pronounced in countries with low performance levels than in high performing countries. A small amount of variance (between 5% and 15%) in persistence is located at the school level. The pattern of correlations of students’ performance level with persistence within and between schools varies between countries; positive as well as negative correlations between performance level and persistence can be found within as well as between schools.