Dr. Rethman’s comments center around four concepts. First is the idea of statistical power, the ability of a study to find a significant difference, if indeed one exists.
The second is sample size, the number of required subjects per group to have adequate statistical power. Power and sample size are highly related. If a large difference between these two groups naturally exists, a study can have the power to demonstrate this difference, even though it has very few subjects.
Conversely, a study involving a large number of subjects will have so much power that it can easily detect even very small differences between groups. In fact, these differences may be so small as to be of no practical significance, even though the study demonstrates statistical significance.
The third concept is the need to define an important difference between groups. This definition has little to do with statistics and everything to do with knowledge of the area of expertise in which the research project is set.
The final concept is that of making inference from a study conducted in a very specific way, using a specific group of subjects to a broader setting and a larger group, and of the role that statistics play in making such inferences.
In the present setting, the reader is most likely interested in making inference from this clinical study to his or her dental practice routines. This process is not a statistical one. As Dr. Rethman points out in his guest editorial in the Journal of Periodontology, statistics are meant to help explain the results observed in a given study.
Statistics in and of themselves do not prove a causal relationship between two events. The reader must examine the study’s strengths and weaknesses, because practical considerations usually make a perfect study impossible. Then considering the study as a whole, the reader must make his or her own inferences.
Dr. Rethman rightfully points out that our study has a limited number of subjects and, therefore, probably lacked the power to find a significant difference between the two groups, if one existed. The conclusion stated in the abstract for the article is meant to refer to this particular study. It is based on the research as a whole, not solely on the statistics. It was based on the clinical observations made while conducting this clinical trial and supported by statistical analyses.
It was not our intention to conclude that the two treatments are generally equivalent, nor are we asking the ADA to certify the two protocols as equivalent. If this was not clear in the original article, we are pleased to correct the situation.
Dr. Rethman also expresses concern over the length of the study. In reality, for a controlled experiment of this type, the length of the study was longer than average.
One only needs to examine the raw data in Table 2 of the article to realize that these two groups of restorations were very similar. For a study with two groups this similar to have adequate power to find a significant difference, 7,169 subjects per group would be required (SigmaStat for Windows, Version 2.03, SPSS Science). As for follow-up, a period of five to 10 years might well be viewed as inadequate.
So while we are in complete agreement that the inclusion of more subjects and a longer study period would have made for a superior study, it simply was not practical to do so. The speed at which new products for operative dentistry are released and accepted into dental practice makes a five-year study with 14,000 subjects obsolete before it can even be finished.
In this age of evidence-based medicine and evidence-based insurance payments, the dental profession may eventually have to decide what qualifies as evidence of such high quality that the profession should base clinical practice routines on it. But until then, we will have to use the best evidence available.
While the number of subjects included in the study is a drawback, there were strengths as well. The strengths of the study are that it was carefully designed and executed, it yielded clinical rather than laboratory data, and the study period was of a good length. Subjects were randomly assigned to create two groups that were equivalent at the onset of the study. Evaluators and subjects were kept unaware of the subjects’ group assignment to avoid bias based on their preconceived expectations of what should happen.
Finally, the methods and materials used were presented to JADA for review by colleagues knowledgeable in this field, and published for all readers to evaluate and critique. By contrast, while expert opinions abound in dental newsletters, in the unsolicited journals that arrive monthly and in the commercially available product review newsletters, there is little information available about what makes the author an expert and how the information was compiled. Also, there is very limited or no review.
Practitioners ultimately will make their own comparison to the other sources of information available to them, but to us the type of evidence gathered from a controlled study and presented in a refereed journal is the best evidence currently available in many areas of dentistry.
