Dr. Colburn’s viewpoint is based on anecdotes of poor clinical performance he observed as an examiner. We don’t know their number or anything about the circumstances that may have contributed to the observations.
Similarly, although as faculty members we might be in agreement with examiners on selected cases, we also could provide anecdotes wherein we have examined patients treated in licensing examinations, and seriously disagreed with the examiner’s pass-fail decision. We cannot depend on anecdotes to determine the reliability or validity of an examination.
Determining reliability, for example, requires analysis of repeated observations, and calculation of the degree of agreement between and within examiners, and then using the magnitude of disagreement to avoid making critical decisions within the margins of error. Determining content validity requires analysis of the degree to which test content represents actual practice, and determining predictive validity requires analysis of postexamination performance of people who pass versus performance of people who fail.
Data on such determinations of reliability and validity for clinical licensing examinations have never been published so that our community of professionals could evaluate them. We know from our study that the results of NERB’s examination were highly inconsistent year-to-year, and its parts were internally inconsistent.
Clinical licensing examinations rely on one-time observations. We know from peer-reviewed, published experimental evidence that one-time observations of clinical performance are highly unreliable. Some of the unreliability is due to examiner variation, but to a greater extent it is due to other uncontrolled sources of variability.1
As clinical researchers who calculate the degree of agreement within and between examiners in trying to relate clinical change to other variables, we also know that multiple, independent assessments are necessary to validate critical measurements and overcome the unreliability of single determinations. One advantage faculty have in evaluating student performance is the opportunity for multiple observations.
We note also that our school performs evaluations of student competency for each of the abilities to which Dr. Colburn refers. All of our students must pass these competency evaluations before graduation. We assume that is true for most, if not all, other schools also, as accreditation requirements now are based on outcomes, not process, and require the measurement of competencies for each student. So how do inconsistencies with the clinical licensing examination happen? They result primarily from the unreliability of single measurements together with the multiple factors that contribute to validity for various purposes.
TOP
REFERENCES
