12th International Ottawa Conference on Clinical Competence edition:12 location:New York City date:20-24 May 2006
In earlier research, the relationship between the ability to self-assessment and an external criterion (such as a parallel assessment by tutors, peers or professors) was studied (Falchikov N & Boud D, 1986; Gordon MJ, 1991; Tousignant M & DesMarchais JE, 2002). In all of these studies a small degree of agreement is found between the external evaluation and the self-assessment. In our research the relationship between the ability to self-assess and an internal criterion is studied: especially the own performances on a knowledge test.
Several authors suggested various “Admissible Probability Measurement Procedures” (Shuford E, Albert A, & Massengill NE, 1966) as operational methods to indentify “metacognitive realism” (Leclercq D & Bruno J, 1993) In our research metacognitive realism has been made operational according to a series of criteria as defined by Nelsson and Narens (1992) .The confidence in the correctness of the formulated answer to an item of a test is not only considered to be “the state of believing that a particular piece of information has been correctly retrieved from memory” (Miner AC & Reder LM, 1994) but also as the conviction that additional processes (analytical and logical) have produced a correct answer to a question.
How accurate can post-graduate trainees define the boundaries of their knowledge?
How well can the candidates differentiate knowledge and distinguish between knowledge that is mastered with certainty and knowledge about which they have doubts?
Is metacognitive realism a dimension different from medical competence?
We are interested in the question to what extend metacognitive realism is related to the results on a knowledge test of which the psychometric characteristics are well documented.
The research on the assessment of medical competence has in the past decades been controlled by the insight that the medical problem solving ability is not generic but content specific (the so called “case-specificity”). Is metacognitive realism domain specific as well?
De result of two major studies are reported in this presentation.
In a first study ( Degryse & Roex 2003) trainees were assessed with different instruments (n=180).
In order to overcome content specificity whilst limiting testing time ,knowledge testing was restricted in a second study to four domains (n=168) (Dory V , Degryse 2005)
Spectral analysis was used as an new innovative method to capture “metacognitive realism” based on a model proposed by (Jans V & Leclercq D, 1999) This researchers studied the ability of adults to discriminate between different levels of uncertainty on a certainty scale and showed that this sensitivity is logarithmical.based.. Based on two formal characteristics of the hemispectrum, the skewness and the kurtosis, a typology of curves was developed (Degryse & Roex 2003)
The relationship between the total test score and the type of curve in the left and the right hemispectrum has been examined in a regression model
Results & Discussion
In this research, no link has been found between the total test score and the results of the spectral analysis. The ability to mark the boundaries of one’s own knowledge in the form of a series of micro predictions on item level does not correlate with the medical competence as measured in the written knowledge test. All this supports the hypothesis that what was described as metacognitive realism can be considered as a separate dimension of medical competence.