It may look like a piece of junk, but that old car probably has a lot of useful life left in it. Even if it’s not a classic waiting to be restored, it is a good source of replacement parts. And when the last door and fender and water pump have been salvaged, the crusher awaits to transform the rusted old hulk into something new and different.

Not so junk science. Useless, dangerous, too toxic even to be recycled, it’s a pure waste product from the outset. What is it? Why is it a problem? And what can we do about it?

My own definition of junk science is a publication that has the tone and trappings of science, but is so fundamentally and demonstrably flawed as to lack any serious claim to credibility. Note that mere errors don’t qualify. Neither does work that you or I may consider obvious, trivial, insignificant or uninteresting. Results based on falsified data are a particularly odious form of junk, not to mention fraud, but junk doesn’t necessarily involve any intent to mislead.

No, junk science is special, in part, because it looks so plausible at a distance but is so easy to recognize up close. Consider some classic categories of the stuff.

One-shot wonders.
Because we deal with living organisms in all their complexity, it’s a difficult matter to sort out real phenomena from random flukes. One patient improves while another deteriorates. Why? We may have our suspicions, but we don’t really know until we have seen enough similar cases to sort out the two or three significant factors from the thousands of irrelevancies. For this reason, it’s a rare study that can draw any valid statistical conclusions from a handful of observations. How many is enough? Those ubiquitous P values provide part of the answer, but you should also apply your clinical experience as a check on whether the number of cases studied seems in line with the scope of the conclusion claimed.

But let me be clear about one thing. Case reports, which may deal with as few as one interesting patient, are a cornerstone of clinical science. They are no less "scientific" for their descriptive, as opposed to statistical, character. History is full of instances in which alert clinicians have pointed the way toward breakthroughs by sharing isolated observations with their colleagues. Case reports become junk only when they try to overgeneralize.

Boutique populations.
No clinical research can be extrapolated to populations that are significantly different from those that were studied. A finding in men can’t be assumed to apply to women; results from Europeans can’t be applied blindly to Japanese; and so on. Researchers with an ax to grind often apply statistical correlations to specially selected populations to draw dramatic but unsupportable conclusions. For example, suppose you surveyed a group of adult cancer patients, and found that 89 percent of them drank fluoridated water. Does that prove a link between fluoride and cancer? Of course not, but there are those who would try to make the link.

Out of control.
Careless or malicious experimental design can virtually predetermine the results of a study. No scientific doctrine is more sacred than that of the control, the fixed benchmark against which a hypothesis is to be compared. Some of the purest junk is associated with bad controls. For example, I once saw a presentation of a weight-loss drug that suggested outstanding performance against a placebo. The results were dramatic, the statistics impeccable. Just one small problem: the placebo was full of calories!

It can be exceptionally difficult to devise a well-controlled experiment, but there’s no other way to get at the truth. If a paper reports on a poorly controlled or, worse, uncontrolled experiment, don’t waste your time on it.

The incredible shrinking denominator.
"A total of 245 implants were placed in 61 patients. ... After three years, 134 of the 145 implants remaining in service were found to be stable, for an overall success rate of 92 percent." Huh? Did I miss something? These patients might not share the author’s idea of "success." Though this example may seem extreme, it’s only slightly exaggerated. When patients (or teeth or restorations) exit an experiment before completion, they need to be properly accounted for. It’s not always necessary to assume that "lost to follow-up" is the same as "failed." Nevertheless, a conscientious scientist will always explain how such lacunae were analyzed so as not to unfairly favor one hypothesis over another.

Proof by endorsement.
Perhaps this one shouldn’t be dignified even as junk science, but there are those who persist in using it. To survey one’s customers, offer them as references and quote them in advertisements is a perfectly acceptable and useful practice. The problem comes when the comments of doctors or patients are converted into pseudoscientific data by means of crude, self-serving surveys. Even when there is no intent to deceive, human nature comes into play to cause a "bias of ascertainment"—which includes, among other things, the fact that people are more likely to respond to surveys when they are either very happy or very unhappy.

If it’s natural, it must be safe.
Remedies billed as herbal, natural, organic, holistic, nontraditional and the like constitute a vast industry, buttressed by a massive amount of junk science (along with, to be fair, a certain amount of legitimate research). Much of the material published in this arena can be easily recognized by one or more of the classic signs already mentioned (lack of controls, unspecified doses and potency, and a great deal of personal endorsement). There’s also a special ingredient unique to this category: the assumption that anything natural is good, wholesome and positive; and that long-standing practice is proof of efficacy. Such ideas can flourish in part because most of these products are marketed as "nutritional supplements" and so evade the scrutiny of the U.S. Food and Drug Administration. Now, there’s no doubt that some traditional remedies work, and we should not blithely dismiss the wisdom of the ages without a good close look. But the world is full of natural things that are unwholesome, and ancient practices that are cruel and harmful, and I see no reason why we should close our critical faculties as soon as the word "natural" is intoned.

Proof by intimidation.
In what has become the commonest form of junk science, all pretense of the scientific method is abandoned in favor of public debate. A strong-willed, articulate advocate for some point of view can speak or write persuasively for that position, blending vivid stories of individual cases, now horrifying, now inspiring, with sweeping allusions to "hundreds of scientific studies." It’s difficult to counter this kind of onslaught by means of statistical arguments and sweet reason. Sadly, this is what the public hears, and increasingly accepts, as science.

Junk science has always been with us, and it’s not going to disappear. But we don’t have to take it lying down. As professionals, we need to develop the habit of translating statistics into plain words, which will go a long way toward demystifying the scientific literature, both good and bad.

Even more important, we desperately need to get our fellow citizens educated in the rudiments of science. Scientific illiteracy is pandemic; people who would be ashamed to admit they had never read Shakespeare will almost proudly state they know nothing about science. No wonder they so easily mistake shiny junk for the real article.