We compliment Dr. Sims and WREB for their continuing and substantial efforts to improve their examination, for recognizing the need to follow established procedures for examination development and for compiling validity and reliability evidence.
Also, we were very impressed that the Board commissioned an excellent external review of its technical work by a nationally recognized scholar. Were we to be conducting the same review, we would have argued for a more conventional survey of practitioners, rather than an expert panel, to assess the knowledge and skills needed for effective practice in the dental field, and to lay the foundation for a content-valid examination. But the WREB validity effort, nonetheless, is commendable.
We do remain concerned about the interpretation of some of the reliability evidence in WREB’s documentation. Those data reflect the ability of examiners to reach the same independent conclusion (that is, interrater reliability) in exercises in simulated clinical situations. We certainly agree that high interrater reliability is important and can be achieved. But that is not the same as the reliability of the examination itself, where the variability that produces low reliability comes, not from the examiners, but from the other conditions of the test, including the substantial variability contributed by examination candidates.
The low reliability of onetime-only clinical examinations, and therefore the test as a whole, cannot be overcome by improving the performance of the examiners (which is probably not even necessary, as indicated in WREB’s data), nor by increasing the number of examiners per observation, as shown experimentally.1
Portfolios, on the other hand, would not only permit double scoring of candidate data by examiners, but it also would permit candidates to repeat some tasks, when useful, and to demonstrate mastery of a wider array of tasks than is possible during a single examination.
We encourage WREB, and other examining agencies, to publish more of their findings in technical reports and peer-reviewed scientific journals, so that concerns about many technical aspects of the examination can be reviewed and, where necessary, the reviews can provide the basis for improvements. We note in passing that it is common in the credentialing field for technical documentation to be prepared and made available to boards, reviewers and other interested people.
TOP
REFERENCES
