Meeske and colleagues⁹ surveyed recent dental school graduates who had completed board examinations and reported that the examinations were not valid assessments of clinical skills and did not reflect dental practice. Formicola and colleagues¹⁰ conducted a concurrent validity study of the North East Regional Board of Dental Examiners’ (NERB’s) examination results and found no statistically significant correlations between NERB’s examination component results and school grades in restorative dentistry (r = – .05), prosthodontics (r = .11) and periodontics (r = – .02). They reported a growing lack of confidence in the profession’s ability to conduct reliable and valid licensure examinations and argued that this absence of statistically significant correlations demands that clinical licensing examinations be brought in line with professional testing standards.

Ranney and colleagues¹¹ compared dental school performance with NERB scores and deduced that the validity of NERB for licensure decisions was questionable. Chambers and colleagues¹² argued that the essential flaw in clinical examinations is the reliance on a "one-shot" sample of a small segment of skills, understandings and supporting values, with the resulting examination being inherently substandard in terms of validity and reliability.

Stewart and colleagues¹³ conducted a retrospective study of the performance of 524 University of Florida College of Dentistry graduates between 1996 and 2003 on the state dental licensure examinations. They divided their students into four quartiles (quartile 1 [Q1], quartile 2 [Q2], quartile 3 [Q3] and quartile 4 [Q4]) on the basis of their graduating grade point averages (GPAs). They found that students in Q1 (those with the highest GPAs) did better on overall state board examination performance and on various components of the examination than did students in Q2, who did better than students in Q3, who did better than did students in Q4 (those with the lowest GPAs). In addition, Stewart and colleagues found that an increasing proportion of students from Q1 through Q4 failed the examination or specific components. However, as they assert, "a dental educator and a dental board examiner would hope that an independent licensure exam would detect those who cannot perform at a level of minimal competence, but would not ‘fail’ those who are competent."^13(p868) An examination of their data suggests that the Florida State Dental Licensure Examination appears to be failing those who are competent. For example, 54.7 percent of those who failed the overall examination were in the top three quartiles in their class, as were 69.2 percent who failed the periodontics examination, 62 percent who failed the clinical Class II amalgam examination and 64.5 percent who failed the laboratory (manikin) examination. It seems reasonable to assume that those who are not competent are predominantly in Q4 of their school GPA ranking, but these data suggest that large numbers of students who do quite well in school are failing the Florida State Dental Licensure Examination. Although there are many psychometric factors (for example, restriction of range, unreliability of school examinations and clinical grades) that may act to suppress the values of the correlation coefficients between performances at school and on licensure tests, the evidence we cite suggests that patient-based licensure examinations lack validity.

The ethical issues associated with clinical examinations have been raised. Buchanan¹⁴ pointed to the responsibility of follow-up care as a serious problem associated with the use of "humans" in clinical examinations. Feil and colleagues¹⁵ conducted a survey of U.S. dentists concerning their experiences with ethical lapses on their clinical examinations. The respondents reported instances of not arranging for follow-up care even though it was indicated (24 percent), instances in which a lesion was created intentionally (8 percent), instances in which premature treatment was provided for the purpose of the examination (17 percent) and instances in which unnecessary radiographs were taken (32 percent).

Jenson¹⁶ argued that the state becomes an ethical agent when it requires that candidates for licensure perform dentistry on patients. As such, the state is obligated to give full information to the patient, to obtain true voluntary cooperation, to prevent exposure of patients to increased risk, to provide oversight while unlicensed dentists practice and to provide follow-up care when adverse outcomes occur. Additionally, Formicola and colleagues¹⁷ and Hasegawa¹⁸ recommended that "live patients" be banned as test subjects on licensing examinations because of the ethical dilemmas created for candidates, the host institutions and dentistry. In addition, both the American Dental Association and the American Dental Education Association have adopted policies that support the elimination of the use of patients in clinical examinations.^19,20

Because of the concerns associated with the use of patients in clinical licensure examinations, we conducted a study to analyze the contribution of the patient-based component of NDEB’s clinical examination to determine its value and validity.

	SUBJECTS AND METHODS

TOP ABSTRACT SUBJECTS AND METHODS RESULTS DISCUSSION CONCLUSIONS REFERENCES

 
Our study included 1,063 candidates who began NDEB’s clinical examination process between January 1996 and November 1999. We followed the candidates—graduates of nonaccredited dental programs—through the final administration of the clinical III component in December 2003. They were required to complete a four-component certification examination to be eligible for licensure in Canada.

The four components of the examination were written, clinical I, clinical II and clinical III. The written component consisted of two paper-and-pencil examinations given in one day during a morning and an afternoon session. The 300 multiple-choice items tested biomedical sciences, general medicine and pathology, pharmacology, periodontics, operative dentistry, endodontics, prosthodontics, orthodontics, pediatric dentistry, clinical therapeutics and oral and maxillofacial surgery. The passing score for this component was 65 percent.

The clinical I component was conducted in a one-day session and consisted of four paper-and-pencil examinations that tested knowledge of and judgment in radiographic interpretation, oral diagnosis, treatment planning and management of patient care. To pass this component, a candidate was required to achieve an average score of at least 65 percent for the four paper-and-pencil examinations, with no score being lower than 55 percent.

The clinical II component tested the ability of the candidate to perform identified restorative procedures (amalgam, resin-based composite, cast gold, ceramometal and provisional restorations) on a simulated patient. Each of the eight procedures was assigned a mark that was converted to a pass or fail result for the component by means of a scoring grid.²¹

The clinical III (patient-based) component required candidates to perform three or four restorative procedures (cast gold, amalgam and resin-based composite restorations) on patients. Each procedure was assigned a mark of pass, marginal failure or failure, and the marks were converted to a pass or fail result by means of a scoring grid.

Candidates were required to pass both the written and clinical I components before taking the clinical II component. Similarly, they had to pass the clinical II component to be eligible to take the clinical III (patient-based) component. Candidates were allowed to repeat the written, clinical I and clinical II components as many times as they wanted. For patient safety reasons, however, they were permitted a maximum of three attempts to pass the clinical III component.

To help candidates prepare for the examination, NDEB provided detailed information including sample questions, scoring criteria, grade derivation grids and lists of reference textbooks to all registered candidates.

We recorded and stored the examination results in a customized database, which provided us the opportunity to make queries and analyze the data. The data included the number of component attempts and results of each attempt for all components. We analyzed the data statistically to ascertain pass rates within and across component parts of the examination.

	RESULTS

TOP ABSTRACT SUBJECTS AND METHODS RESULTS DISCUSSION CONCLUSIONS REFERENCES

 
Clinical I component. Of the 1,063 candidates who attempted the clinical I component, 835 (78.6 percent) passed. The 228 candidates (21.4 percent) who did not pass were not eligible to proceed to the clinical II component (Table 1). A total of 565 (53.1 percent) of the candidates passed on the first attempt. Of the 498 who did not pass, 334 made a second attempt, and of these 197 (58.9 percent) passed, bringing the cumulative pass rate after two attempts to 71.6 percent. Some of those who did not pass after two attempts completed a third, fourth or fifth attempt.

Clinical III component. Of the 660 candidates who passed the clinical II component, all but one attempted the clinical III component. After three attempts, 637 (96.7 percent) candidates passed. Among the 22 (2.1 percent) candidates in the study (N = 1,063) who did not pass, 12 (1.1 percent) failed the maximum allowable three attempts, while 10 (0.9 percent) chose to not continue after failing the first (n = 7) or second (n = 3) attempt, primarily because they enrolled in an accredited dental program.

Overall. Of the 1,063 candidates in the study, 637 (59.9 percent) successfully complete all of the components and were certified. Failure to successfully complete the clinical I and clinical II components screened out 228 (21.4 percent) and 149 (14.0 percent) of the total candidates, respectively (Table 2). Failure to pass the clinical III component prevented 22 (2.1 percent) of the total number of candidates from being certified. Of these 12 (1.1 percent) failed the component three times (the maximum allowable number of attempts), while 10 (0.9 percent) chose not to continue to take the examination primarily because they had enrolled in an accredited dental program.

	DISCUSSION

TOP ABSTRACT SUBJECTS AND METHODS RESULTS DISCUSSION CONCLUSIONS REFERENCES

The continued use of the patient-based component would not justify the associated costs or address the ethical concerns.

Twenty-six candidates who passed the clinical I component did not proceed to take the clinical II component. Some of these candidates had not passed the written component and, therefore, were not eligible. Others elected not to continue in the examination process for various reasons, including enrolling in an accredited dental program.

The pass rate for the first attempt on the clinical II component was 47.8 percent and ranged between 37.2 and 54.4 percent for subsequent attempts. This fairly stable pass rate for subsequent attempts may be due in part to the same factors we outlined for the clinical I component.

When first established, the purpose of the written, clinical I and clinical II components was to screen out candidates who were not adequately prepared to take a patient-based examination, thus avoiding compromising patients’ safety. Primarily because of time constraints, these components were not designed or considered to be comprehensive evaluations of all of the content areas in dentistry. The subsequent patient-based component was considered to be the component with the highest fidelity and, therefore, the component that ultimately would identify people who were not competent and prevent them from entering practice.

NDEB adopted a sequenced examination process with conjunctive scoring because it concluded that it would be unethical (and legally unadvisable) to allow candidates to participate in the patient-based component without some verification of their knowledge and clinical skills. The sequential and conjunctive nature of the examinations makes it likely that some candidates who would otherwise have failed the patient-based component would have been screened out previously.

The results of our study confirmed that the candidates who had passed the clinical I and clinical II components were prepared to participate in a patient-based component, as 96.7 percent of those taking the clinical III component passed within three attempts. Anecdotal information from examiners and staff who participated in the clinical III component indicate that they generally considered it to be a fair and valuable component that was an important measure in the protection of the public. They also have commented that failing candidates frequently demonstrated a significant lack of clinical skill and judgment.

Nevertheless, almost all of the candidates who failed the clinical III component on their first attempt subsequently passed on their second or third attempt. This occurred despite the fact that the majority of the candidates were unlicensed dentists living in Canada who were not able to prepare for their next attempt by practicing on patients.

The high pass rate and, in particular, the high second and third attempt pass rates for the clinical III component suggest that this patient-based component did not contribute to the overall validity and did not prevent candidates from obtaining certification. Furthermore, the continued use of this patient-based component would not justify the associated costs or address the ethical concerns. In contrast with the ability of the clinical I and clinical II components to screen out approximately 40 percent of candidates, the clinical III component prevented only 1.1 percent of candidates from being certified as as a result of their failing the component in the allowed number of attempts.

Although the written, clinical I and clinical II components appear to have effectively screened out candidates unprepared to take a patient-based component, we cannot assume that candidates who passed these components were competent to practice, as they were not designed to be comprehensive examinations.

Therefore, after analyzing the results of the certification examination process and after an intense and lengthy consultation process, NDEB replaced the examination process for graduates of nonaccredited dental programs with the requirement that they had to complete an accredited qualifying/degree completion program followed by successfully completing NDEB’s written and OSCE components. This resultant certification process was adopted by NDEB and all provincial dental boards as the most effective method of ensuring protection of the public, as it included the benefits of completing a comprehensive, accredited educational program followed by an independent verification of competency (NDEB’s written and OSCE components), which is identical to the verification required for graduates of accredited undergraduate dental programs.

	CONCLUSIONS

TOP ABSTRACT SUBJECTS AND METHODS RESULTS DISCUSSION CONCLUSIONS REFERENCES

 
The results of our study on the validity and reliability of patient-based examinations are consistent with those of other reports^6,8,11 that concluded that these examinations are not effective methods of evaluating competence for practice.

Generalizing the results of our study to a different population of candidates or to an examination that does not use conjunctive scoring may not be valid. Nevertheless, the results of our study should cause organizations administering high-stakes, patient-based examinations to assess whether the patient-based component contributes to decision making.