A recent report found no differences in class rank or grade point average, or GPA, between graduates who failed and those who passed the restorative section (amalgam and composite restorations) of a clinical examination given by the North East Regional Board of Dental Examiners, or NERB.² This report also found a wide distribution of class ranks for both those who failed and those who passed NERB. At the same time, there was a difference in academic performance between students between those who passed and those who failed a NERB exercise on a manikin, though again the distribution of class ranks among failing and passing graduates was highly dispersed.

The results of that study added to questions about validity of licensing examinations for making decisions about licensure and increased concerns about the ethics of irreversible procedures on patients in those tests.² Those results, however, were for a single year only, which might have been unrepresentative of the general results of NERB’s examination. Therefore, in the current study, we assessed the relationship between dental students’ performance in dental school and performance on NERB’s clinical examination by assessing results over nine years.

	METHODS

TOP ABSTRACT METHODS RESULTS DISCUSSION CONCLUSION REFERENCES

 

Since the North East Regional Board of Dental Examiners uses a conjunctive scoring method, the overall result was failure for those who scored below 75 on any of the sections.

We studied the results of NERB clinical examinations that were performed in May of the years 1994 through 2002 at the Baltimore College of Dental Surgery, Dental School, University of Maryland. We analyzed data representing the 835 doctor of dental surgery graduates of the school during that period. We determined the class rank for each graduate within each class based on his or her overall GPA, and then we normalized the class rank for comparability among classes by converting it to a percentile. We used the results of each graduate’s first time taking NERB’s clinical examination (as reported to the school by NERB) from each of the examinations major sections:

– the dental simulated clinical exercise, or DSCE (written);

– the restorative clinical exercise, or RESTOR;

– the simulated patient treatment (manikin) clinical exercise, or SIM PATIENT;

– the periodontal clinical exercise, or PERIO.

Since NERB uses a conjunctive scoring method, the overall result was failure for those who scored below 75 on any of the sections. NERB reported no results for 235 of the 835 graduates in the year of their respective graduations; this meant that those graduates either did not take the examination or did not provide NERB with written permission to release the scores.

We conducted statistical analysis of the data in three parts. We used logistic regression to investigate two questions: were the passage/failure rates consistent over the nine years? What was the diagnostic value of the clinical tests as indicated by receiver operating characteristic, or ROC, curves for determining the quality of the dental students? A clinical test in dentistry is similar to a diagnostic test in medicine. The goal of any diagnostic test is to separate abnormal results, which indicate disease, from normal results, which indicate health. The goal of a licensing examination is to separate people who have the knowledge and ability to practice from those who do not. ROC curves provide a tool for evaluating such diagnostic tests.¹⁴ They evaluate the diagnostic quality of a test by presenting a visual representation of sensitivity and one minus specificity for a range of values of the test, which in our case was used to diagnose class rank percentile. For example, one can determine from the curves if the test has high sensitivity and specificity for detecting students with low class rank. The diagnostic value of the test is measured by comparing the distance of the curve from a diagonal across the chart. The diagonal is representative of a test with no diagnostic ability; thus, the further the curve is from the diagonal, the better the diagnostic ability of the test.

For the second part of the analytical strategy, we used the Fisher exact test and we estimated a statistic to determine if the RESTOR section and the SIM PATIENT section of the clinical examination produced similar results. In other words, we tested the agreement between those two sections of NERB’s examination.

We used analysis of variance, or ANOVA, to test whether the mean class rank percentile was similar or different among the three groups of graduates (those who passed the test, those who failed and those for whom we had no reported results). We then used the Tukey test for multiple comparisons to test for differences between each pair of groups. We used the ANOVA models separately for the four licensing sections being evaluated and for the composite overall passage/failure rate.

	RESULTS

TOP ABSTRACT METHODS RESULTS DISCUSSION CONCLUSION REFERENCES

 
Figure 1 depicts the year-to-year variation in failure rates over the nine years of this study. The overall failure rates, the RESTOR section and the SIM PATIENT section each varied significantly over time (P < .0001), whereas the year-to-year failure rates for the PERIO section and the DSCE section did not (P > .1 and .5, respectively). Additionally, as shown in Table 1, the failure rates of the RESTOR and SIM PATIENT sections were inconsistent with one another over the nine years—that is, their passage/failure rates varied independently from one another. When we compared the students who passed or failed those sections, we found no agreement between them. A total of 12 percent of those who failed the SIM PATIENT section also failed the RESTOR section, and 10 percent passed the SIM PATIENT section but failed the RESTOR section. The statistic for the comparison was 2 percent, which was lower than its standard error of 4 percent.

View larger version (38K): [in this window] [in a new window]   Figure 1. Percentage failure of the North East Regional Board examination and its sections, by year. RESTOR: Restorative clinical exercise. SIM PATIENT: Simulated patient treatment (manikin) clinical exercise. PERIO: Periodontal clinical exercise. DSCE: Dental simulated clinical exercise (written).

View this table: [in this window] [in a new window]   TABLE 1 COMPARISONS OF GRADUATES WHO FAILED AND PASSED THE RESTOR* AND SIM PATIENT SECTIONS OF THE NERB EXAMINATIONS.

The ROC curve for evaluating class rank by failure of the NERB clinical examination (overall failure) is shown as Figure 2. It indicates that the examination was not a good diagnostic tool for that purpose. The curve is close to the diagonal, and there is no point on the curve that has high sensitivity and an acceptable false-positive rate (one minus specificity). Each of the ROC curves for the sections involving on-site evaluations by examiners was similar to the curve for overall results. To illustrate, the ROC curve for the NERB’s RESTOR section is presented as Figure 3. The analogous curves for the PERIO and SIM PATIENT sections were nearly the same, so we did not include illustrations for them. Only the DSCE section offered the possibility of achieving a 90 percent sensitivity at less than a 60 percent false-positive rate (Figure 4, page 1151). At 80 percent sensitivity, the DSCE section had about 30 percent false-positives, and at 70 percent sensitivity, it had about 15 percent false-positives.

View larger version (45K): [in this window] [in a new window]   Figure 2. Receiver operating characteristic curve for evaluating class rank, by failure of the North East Regional Board examination.

View larger version (44K): [in this window] [in a new window]   Figure 3. Receiver operating characteristic curve for evaluating class rank, by failure of the restorative clinical exercise section of the North East Regional Board examination.

View larger version (48K): [in this window] [in a new window]   Figure 4. Receiver operating characteristic curve for evaluating class rank, by failure of the dental simulated clinical exercise (written) section of the North East Regional Board examination.

	DISCUSSION

TOP ABSTRACT METHODS RESULTS DISCUSSION CONCLUSION REFERENCES

The significant variation in certain failure rates from year to year suggests that either the tested abilities of graduates were different from year to year or that the NERB examination itself was different from year to year. Concern about this variation in the test is compounded by the inconsistency between the results of the SIM PATIENT and the RESTOR sections within the NERB examination. Although there are differences in some skills tested in the SIM PATIENT and RESTOR sections (for example, the SIM PATIENT section uses a typodont for an extra-coronal procedure and the RESTOR section requires clinical decision making for an intracoronal procedure in a patient), they are the most closely related of the four components of the NERB examination. The final products involve preparations and physiologically contoured restorations that use similar hand-eye coordination skills. Consequently, one would expect some measure of agreement between them. But with a statistic that was essentially zero, the only possible conclusion is that the tests fail to validate each other. The finding that failing more than one section of the examination was a rare occurrence strengthens this conclusion. These findings support a hypothesis that the difference in failure rates over the nine years was related to inconsistent evaluations by the clinical evaluators, not to variation in the abilities of the graduates over those years. The hypothesis also is supported strongly by the lack of variation over time of the PERIO and DSCE sections of the evaluation. Whereas NERB and other clinical testing agencies do strive for intraexamination reliability by standardization exercises for examiners, the results of our study indicate that the interexamination reliability (year to year) is not good and that the examiners are not consistent among the different sections of the NERB. We are aware of no other published analysis of these types of variations in clinical dental examination results.

This is not to say that the standardization exercises are without value. Standardization should reduce variation due to measurement error. Standardization for examiners, however, does not ensure that the overall test is valid, or that other, even larger, sources of variation are controlled. In fact, variation attendant to the use of nonstandardized patients as part of the examination can be substantially larger than variation attributable to measurement error, thus reducing or destroying the reliability of the clinical test.

Decisions for licensure should be based on tests that are both valid and reliable. If the variability found in our study is representative of tests in other licensing jurisdictions, decisions across the nation about licensure are being made by licensing authorities on the basis of observations of clinical testing agencies that are suspect for reliability and validity. There is no question that the dental examiners for these testing agencies are dedicated people who take time from their practices or other professional and personal pursuits to conduct the examinations for the betterment of the profession and protection of the public. Despite their efforts, however, the data in this report indicate that the NERB examiners are not likely to accomplish their goal of eliminating unqualified people from licensure. Over the time that NERB has reported results as those who "availed themselves of all opportunities to pass the NERB Clinical Examination in Dentistry," 100 percent of the graduates in our study passed (E.H. Hall, director of examinations, North East Regional Board, written communications, Jan. 23, 2002, and Jan. 15, 2003). NERB’s failure to reach the same conclusion on first examination came at the cost of denying licensure to competent graduates for some period during a time when their educational debt burden is at an all-time high.

Over a nine-year period, there was no significant difference in class rank percentile between those who passed the RESTOR section of the NERB examination and those who failed it. This indicates that a one-time evaluation by NERB examiners of restoration preparation, caries removal, and placement and finish of amalgam and composite restorations essentially does not relate to the quality of the respective students as determined by the dental school faculty. This finding is in agreement with a previous report from a single year’s results from an examination given by NERB.² As the faculty’s determinations are based on multiple observations, and validity of decisions is improved by use of multiple observations,¹⁵ the usefulness of the NERB examiners’ determination that a graduate lacks competence in restorative dentistry is questionable. On the basis of the data from our study, one can conclude that the state boards of dental examiners should question the clinical licensure examiners’ conclusion in that regard, and take more seriously the determination of the faculty. To assure the public that there is not a conflict of interest for the faculty in determining qualification for practice, perhaps those making the licensure decision should take both the faculty’s observations and the observations of an independent third party into account. But based on the results of our study, a decision should not be based solely on the determination of the examining agency, as the decision would in that case lack sufficient validity.

From the data, NERB’s use of conjunctive scoring clearly elevated the failure rates by more than double the rate of any of the examination’s contained sections. In selecting conjunctive scoring, NERB argues that a passing mark in each section is necessary to ensure protection of the public by independent evaluations of competence for each part of the examination that they determine to be important.¹⁶ If reliability of the examiners’ determinations was good, that assertion would be plausible. However, it is not realistic to accept that argument, as reliability from section-to-section was nonexistent when we evaluated it over a nine-year period, and major sources of variation that can contribute to a failing score remain uncontrolled. It would appear, in fact, that the conjunctive scoring method decreases the reliability of the pass/fail decision at the level of the examining agency. Therefore, it also decreases the validity of the decision at the level of the licensing authority if the licensing authority accepts the agency’s evaluation without considering other factors.

Even for the PERIO and SIM PATIENT sections of the NERB examination for which mean class rank percentiles did differ significantly between pass and fail results, those differences were not large; they were only 45 or 46 versus 58, a difference of 12 to 13 percentile ranks out of a possible 99. So while these differences were statistically significant at P < .05, it is unlikely that they were significant in terms of validity of decisions made on the basis of them.

Our conclusion that NERB’s clinical examination lacks reliability as a requirement for licensure is supported further by the ROC curves produced from the data in our study. The ROC curves demonstrate that if the intention is to detect the poorest performers in the graduating classes, the clinical tests do not do the job. The examining community has asserted that only a small percentage of graduates (perhaps 2 to 3 percent¹⁷) should be prevented from obtaining a license to practice through the clinical testing process. It seems reasonable that the worst 2 to 3 percent of the graduates might be found in the lower portion of the dental school class as determined by the dental school faculty. But our ROC curves showed that NERB’s clinical tests could not do much better than a random possibility of making that determination. Most people who failed the examination on their first try did not reliably deserve to. And at least for those years for which we have the relevant report from NERB, all the graduates who persisted in taking the examination after failing the first time did pass within the same year.

The clinical examinations did not provide validity for making the licensure decision.

Over the nine years we studied NERB’s DSCE section, it had a 33 percent rank differential between candidates who passed or failed, which was between double and triple the differential for the clinically evaluated sections. Its ROC curve also indicated that it came closer to being diagnostically useful for academic performance than any other section or the overall results of the NERB examination. We expect that a substantial part of the reason for this is because the uncontrolled variation attendant to use of human subjects (patients) in the RESTOR and PERIO sections, and the subjective determinations made in those sections and the SIM PATIENT section are not present in the DSCE section. It also is possible that part of the reason is that the DSCE section is more analogous to the type of grading and ranking most commonly encountered by students in dental school.

Most, if not all, of the jurisdictions that use the NERB examination also require a passing score on Parts I and II of the National Board Dental Examinations. In addition, some dental schools, including the school that was the source of data for our study, require that a student pass that examination before graduation. The natural question is whether passing both the National Board Dental Examination and NERB’s DSCE section is a reasonable requirement for licensure. A comparison of NERB’s DSCE section and Part II of the National Board Dental Examination performed at the request of the ADA House of Delegates in October 1998 reportedly concluded that they measure different things.¹⁸ That comparison could not determine, however, whether one examination provided more useful information than the other for purposes of the licensure decision, or whether either examination would identify the same people as having passed or failed. A direct comparison of students’ in-school performance on Part II of the National Board Dental Examinations with their performance on NERB’s DSCE section showed that the results from both examinations essentially were the same.¹⁹

While our current study improved on previously published data by using results over a number of years, it still was limited to one school, meaning also that it was limited to that school’s educational program and its facility as a NERB examination site. It would be useful to conduct similar analyses of data from several schools together and from different examining agencies.

	CONCLUSION

TOP ABSTRACT METHODS RESULTS DISCUSSION CONCLUSION REFERENCES

 
Over a nine-year period, the NERB examination results of graduates from one dental school failed to be a good measure for detecting the quality of those graduates as determined by the dental school’s faculty. The sections of the NERB examination that were dependent on examiner observations were less able to make a distinction between good and poor class rank than was NERB’s DSCE section. The interexamination reliability of the NERB examination was low, as indicated by the high year-to-year variation in the clinical examination results and the fact that different sections of the examinations were not able to validate each other, while the results from the DSCE section did not significantly vary from year to year. The clinical examinations did not provide validity for making the licensure decision, bringing into question the ethics of using invasive and irreversible procedures on patients as a part of the dental licensure examination.