Mr. Hammond comments on the currency of some references in my article. He also notes that all testing agencies are not the same, and the cost of a portfolio approach may be unacceptably high. Each of these arguments is correct in a very limited sense, but none has general merit.
Psychometrics is not a field where the science changes regularly, as it does in the biomedical sciences. I know of no relevant differences in the AEDA, APA, NCME or AADE publications Mr. Hammond references and the ones I cited. He does not identify any.
State and regional testing agencies certainly differ in their pass rates. That does not mean that they differ in their reliability. The reason Mr. Hammond could not reproduce the calculation for reliability of the testing agency he works for from the formulas in my article is that there is no way of knowing how many candidates who sit for those examinations ultimately pass some initial licensure examination. Some candidates never pass a particular examination because they have passed one or more alternatives. I used the American Dental Association’s national estimate to avoid this problem.
There are two possible approaches to estimating individual state or regional examination reliability. One is to apply the national average to each examination system. This would be flattering to states such as Delaware, with its typical 100 percent pass rate. The best candidates in the nation would seem to be drawn to practice there based on this assumption. Nevada has recently had pass rates under 40 percent, so the assumption about the national pass rate applying equally seems untenable without suspecting that Nevada is especially attractive to the under-qualified.
An alternative is to assume that the ultimate pass rate in each examination system is proportional to its initial pass rate. On this assumption the reliabilities for the four regional examination agencies range from r = .37 to r = .45 (compared to my national estimate of r = .40). This sensitivity analysis shows that Mr. Hammond’s concern about differences across initial licensure examination systems is theoretically correct, but practically unimportant. The Western Regional Examining Board is the easiest to pass, and has the lowest reliability.
It is correct that improving initial licensure examinations administered by state or regional examining boards to an acceptable level of reliability and validity is too costly to be practical. This program is not, however, beyond the means of dental schools, and there may be other approaches, such as using provisional licenses. The overall fiscal impact of initial licensure would decrease under a portfolio system. (The only exception to this economic benefit would be the regional testing agencies, which would suffer financially.)
If state boards exercise their responsibilities for defining initial competency to practice, establish psychometric standards, and monitor data collection (as I suggest they do), they will retain and strengthen the independent determination of who is qualified to begin practice (as I feel they must). They will be free to delegate data collection to any party that can provide reliable and valid data, not just regional testing agencies or state-administered testing agencies that are constrained to use an indefensible one-shot approach.
