Scientific Criticism By and For the Scientifically Untrained

As a general dentist practicing for some thirty-five years, I have, over the past several years, come to question first the therapies of medical science that have impacted my own health, then the processes by which my profession reaches its therapeutic recommendations, and finally more broadly the way scientific method is utilized within the healing arts.

I'm angry at what I have found.

As a case in point, let me refer to Dr. Fineberg's statement from the United States Institute of Medicine. This statement was in defense of the IOM's report on the safety of vaccines and their suspected link to the rise in autism. Of special concern were those vaccines containing thimerosal, a mercury-laden preservative. Dr. Finberg concludes his report with the statement, "The Institute of Medicine and the National Academies have full confidence that the Immunization Safety Review Committee operated independently, impartially, and in accordance with the highest scientific standards."

The vaccine safety issue has engendered a great deal of dissent by folks well qualified to disagree. Dr. Andrew Wakefield is one. US Congressman Dr. Dave Weldon [A .pdf file] is another. But I find that following their arguments requires more technical and scientific expertise than I possess. Since this and many other medical issues directly impact our health and continued existence on this planet, it is important for the less erudite among us to be able to form worthwhile judgments about scientific debates. I believe we can all do that. You don't have to be a rocket scientist to figure out that the damn thing blew up on the launch pad. (Why it blew up is another matter.)

Intrigued by the promise of Dr Fineberg that all was above board, I proceeded to read the IOM report - (well as much of it as I could stomach.) I submit that the report is clearly and irredeemably flawed by its own definitions and stated rules of conduct. One does not need to be able to weigh the merits of the evidence presented to be able correctly to label the entire process as worthless.

That is the conclusion I have reached according to the "Bogus Science Detector" (BS detector for short) that has assembled itself within my cranium over the last few years. Should any reader be able to point out flaws in the operation of this device – I have no doubt that such flaws exist – I would be delighted to hear from them at ohapomm@bdmclean.com (That stands for "Offer Him A Piece Of My Mind.")

Let me first describe the evolution and workings of my BS detector. If you like it, you can assemble one for yourself, absolutely free, within your own cranium!

Having thus established a functioning BS detector, we can check out Dr. Fineberg's committee's report and see if the detector starts to clang.

The scientific knowledge enabling the construction of my BS detector comes in part from three great thinkers:

1. Thomas Huxley

Thomas Huxley was the progenitor of a whole gang of illustrious Huxley's who said, well over a hundred years ago,

"The great tragedy of science – a beautiful hypothesis slain by an ugly fact."

Stripped of the emotional words, tragedy, beautiful, slain, and ugly, we derive the essential meaning that a hypothesis must explain all the relevant facts. Add back those emotionally charged words and we taste Huxley's sarcasm.

Scientists and their financial underwriters seem to love their hypotheses; anything that might threaten them is necessarily "ugly".

2. Max Planck

Now consider the words of the Nobel Laureate who discovered the fundamental constant of nature:

"A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it."

Planck did this work in 1900, and his discovery of "Planck's constant" was the first hint in physics of what later became quantum mechanics. Plank had no idea. He thought that by inserting a constant into his equation he was fudging his math to get a right answer and expected that someone would iron out the kinks in his theory or replace it with a better one.

Perhaps they will. We can realize two "rules" of science based on this:

Rule 1. It's OK to be wrong.

Rule 2. Most scientists don't believe Rule 1.

This universal, powerful scientific inertia, as it occurs in the medical field, must inevitably mean that some (much?) of the "wisdom" driving the medical advice you receive is outdated or downright wrong.

In fact, it is the duty of medical (and other) scientists, not to defend a current and revered theory, but to seek to destroy it; not to ignore data they dislike but actively and aggressively to seek them out. There is clearly a long history of science, scientists and those who profit by applying the technology that science inspires doing less than what we are entitled to expect of them.

3. Karl Popper

Sir Karl Popper received a knighthood for his contributions to philosophy – particularly with respect to the philosophy of science. He has pointed out that one can never prove a hypothesis to be true. But one can readily demonstrate that a hypothesis is false. The fact that one can amass an enormous amount of evidence in support of a hypothesis surely must mean one of two things:

(1) The hypothesis is correct in every respect.

(2) It is not, but no one has discovered the flaw(s) – yet.

This means that the only way science can truly advance is by seeking flaws in its hypotheses. Anyone who can demonstrate such a flaw, anyone who can produce a fact or piece of evidence, which cannot be explained by an existing hypothesis, should be welcomed. Why, then, are such facts regarded as ugly? Perhaps we should ask Dr. Olivieri, for instance, why she was fired from her research position at Toronto's Sick Children's Hospital.

Let us therefore build some triggers into our BS detector.

"The great weight of evidence" is interesting, impressive and often convincing. But doubling the size of the evidence pile does not make the hypotheses any more "true!" Yet the very phrase "great weight of evidence" implies that there is a lesser weight of conflicting evidence. What ever has happened to that little pile of ugly facts?

The answer that one hopes to find is each and every ugly fact has been examined for merit or lack of it. Can one demonstrate, error, fraud or sloppiness of technique? Can others reproduce the evidence? If not, were the replication experiments conducted fairly and honestly?

Such activity and debate is truly what science should be about.

Of course, if all the conflicting evidence were properly shown to be invalid, there would be no collection of such data in need of being "overwhelmed" by "the great body of evidence."

Thus the phrase "the great weight of evidence" in itself implies an absence or at least a shortfall of such activity. My BS Detector starts to grumble. Ah, I hear you cry, there is always some flake with a nutty notion. True. There may not be time or resources to check out all of them. What about the Ph.D.'s and others who dare to dissent? Are they flakes too? Why are they so often "dismissed?" I choose that last word carefully.

At its worst, the "weight/body of evidence" arguments translate as follows: If a fact and the hypothesis it may engender run contrary to an "accepted" hypothesis, the former should be ignored as unworthy of attention. In other words, if it's not in accord with "the great weight of evidence," it's wrong.

That last restatement of the "weight of evidence" argument is so absurdly antithetical to the scientific principle that the hypothesis must account for all the data, no sane scientist would support it. That, I suppose, is why they couch their arguments in fuzzier terms. That way they don't have to confront the real meaning of what they are saying.

Apart from dismissing those who dare to dissent – (shooting the messenger, as it were) – science has devised various other filtering mechanisms to admit only data considered legitimate of consideration. (But as I understand Popper, any hypothesis is fair game. He prefers the certainty of eventually being able to disprove a hypothesis to the impossibility of ever being able to prove one.)

Whatever its merits, the peer review system serves as a strong, anonymous censorship panel, to which there is no right of appeal. That is truly a peculiar institution to set up where free exchange of ideas is supposed to occur. Peter Duesberg, Ph.D., a much-beleaguered opponent of the HIV-AIDS connection puts it this way: [A large .pdf file]

"The corporate equivalent of academia's 'peer review system' would be to give General Motors and Ford the authority to review and veto all innovations by less established carmakers competing for the consumer."

How would you feel, for instance, if you were denied any knowledge of who the Supreme Court Justices were, by what merit they reached office, or how they ever voted?

Think about it. If pursuit of truth through science were paramount, one would design a system meant scrupulously to unearth "ugly facts" and demonstrate their merit or lack thereof. Why would one entrench anonymity and lack of right of appeal in that system? Such features, however, would ideally serve the agenda of a body whose purpose was to defend the status quo.

Even the Cochrane Collaboration can find little evidence to justify the system. They conclude:

" At present there is little empirical evidence to support the use of editorial peer-review as a mechanism to ensure quality of biomedical research, despite its widespread use and costs."

What is the Cochrane Collaboration? This is a body formed to support clinically the practice of evidence-based medicine. The latter is an attempt to organize thought processes involved in evaluating scientific data – truly a worthwhile objective.

One principle of evidence-based medicine is that clinicians should carefully evaluate whether conclusions in papers they read are supported by data. Studies nowadays are analyzed with such advanced statistical methods that it truly requires a statistician to evaluate them. Few physicians are statisticians and even if they were, few have the time to evaluate studies in the way required. To the rescue comes the Cochrane Collaboration, a group of folks with the credentials properly to evaluate these studies.

In practice, however, there are many problems.

If I had to pick one, it would be the one exemplified in the August 2003 edition of Journal of the American Medical Association.

Researches in Copenhagen analyzed 167 random-controlled drug studies from the Cochrane Collaboration. They showed that when apparently unbiased studies were funded by for-profit organizations it was over three times more likely that the experimental drug was recommended as treatment of choice than when funds were supplied by nonprofit organizations (16% vs. 51%.)

The kicker is the last line in the abstract's conclusion:

"Readers should carefully evaluate whether conclusions in randomized trials are supported by data."

Since the purpose of the Cochrane Collaboration is to spare clinicians the necessity of doing precisely that, what good is it?

So when I hear an evocation to "The Great Weight of Evidence" or reverence for the Peer Review System or Evidence-Based medicine by BS detector becomes sensitized. Then I read on to see what is done with such devices. When I discover that they are used as licenses to dismiss, discard or ignore evidence investigators don't like, my BS detector clangs so loudly that it becomes difficult to continue reading. Usually I stop, as I did in the case of the IOM report. It is possible, even probable, that the report makes some cogent arguments. But since I know that they are based on incomplete evidence, I could not put much store by them.

So what, specifically, is wrong with the IOM report? To my BS Detector, this part:

"Published and Unpublished Data

Published reports carry the most weight in the committee's assessment because their methods and findings are laid out in enough detail to be assessed. Furthermore, those published works that undergo a rigorous peer review are subject to comment and criticism by the entire scientific community. Thus the committee generally cannot rely heavily on unpublished data in making its scientific assessments (regarding either causality or biological mechanisms) because they usually lack the commentary and criticism provided by peer review and must therefore be interpreted with caution."

What a lovely "Catch 22." The peer-review system won't publish conflicting data. Nor is it obliged to say why. Nor is there a right of appeal to that system. Being unpublished it lacks the criticism (or support) it might otherwise have garnered.

Lacking exposure to criticism, it must be no good anyway.

No doubt the peer-review system prevents a lot of garbage from getting into journals. But how many babies are thrown out with the bathwater? We will never know. The bathing and subsequent dumping were never reported.

"The committee also relies on editorial and peer-review procedures to ensure the disclosure of potential conflicts of interest that might be related to sources of funding of the research studies. The committee does not itself investigate the sources of funding of the published research reports it reviews, nor do funding sources influence the committee's interpretation of the evidence."

Gee, that's reassuring. Trust the system when it supports your agenda; but when that agenda is likely to be disrupted, ignore the principle of evidence-based medicine that requires one to take into account how the funding of a study might corrupt its outcome.

"Unpublished data and other reports that have not undergone peer review do have value, however, and are often considered by the committee. They might be used, for example, in support of a body of published, peer-reviewed literature with similar findings. If the committee concluded that the unpublished data were well described, had been obtained using sound methodology, and presented very clear results, the committee could report, with sufficient caveats in the discussion, how the unpublished data fit with the entire body of published literature."

So as long as the unpublished data is "in support of a body of peer-reviewed" etc.- (read 'the great weight of evidence') – the committee will take the trouble to consider it. Nothing could be a more perverse diversion from true scientific inquiry! Anyone interested in uncovering the truth, in gaining new knowledge, would know that amassing more evidence in support of what we think is true, while interesting, contributes little. Only the "ugly facts" can show that what they think is true is actually in error. But they apparently don't want to consider those. To that end…

"Only in extraordinary circumstances, however, could an unpublished study refute a body of published literature."

So while Dr. Fineberg and his committee dodge and discount evidence they don't like, although I may be lacking the ability to evaluate the data in question, my BS detector and I are as confident about the integrity of their findings as we are about the pronouncements of any kangaroo court anywhere.

Brian D. McLean, B.Sc., D.D.S.

Barrie, Ontario, Canada

ohapomm@bdmclean.com