This slide presentation, and narrated video version, look briefly at the danger of relying on individual clinical experience rather than formal scientific research in making medical decisions.

Slide Presentation

Clinical Trials or Clinical Experience?

This slide presentation, and the narrated video version, are part of a talk given at the annual convention of the American Veterinary Medical Association. The subject was whether or not one can apply the approach of evidence-based medicine to complementary and alternative medical practices. The presentation reviews some of the philosophical principles underlying both approaches and discusses potential areas of conflict between them.

A slightly more detailed examination of the subject will also be available as a commentary in the upcoming August 15th issue of the Journal of the American Veterinary Medical Association.

McKenzie, BA. Is complementary and alternative medicine compatible with evidence-based medicine? Journal of the American Veterinary Medical Association. 2012;241(4):421-6.

Slide Presentation (Powerpoint):

Can Complementary and Alternative Medicine be Evidence-Based Medicine?

The direct quotations from Dr. Eisen’s web sites were temporarily removed pending the outcome of a DMCA copyright complaint. Since no legal action was taken during the 14-day waiting period mandated under the DMCA, the original quotations have been restored.

Most of the individuals I write about who are promoting questionable veterinary medicine, or outright nonsense, are veterinarians. Such individuals have some claim to legitimacy and authority in recommending veterinary treatments, and so their voices tend to carry more weight than those of most lay people. This isn’t always a good thing since veterinarians are no less prone than anyone else to getting bad ideas stuck in our heads or to being misled by all the features of the world and our own brains that so easily fool us. Experts, that is individuals with recognized credentials and formal training in a specific area of medicine, do tend to have more factual knowledge in that subject. But they are not immune to making mistakes, so as always it is the ideas and the evidence that must be given the greatest way, not the authority of the individual.

Because advice from veterinarians is more likely to be trusted, even when it’s bad advice, I focus on these sources when pointing out why certain ideas, practices, or products are not supported by good evidence and science. However, once in a while I run across a particularly egregious example of misleading a negative advertising for alternative medicine that I feel obliged to challenge even though the source is not a veterinarian. Here’s a quick example of why I noticed this one:

If your dog has been given the heart-breaking diagnosis of terminal cancer… Just About EVERYTHING Your Vet Told You Is WRONG!

Don’t Let Deadly Mistakes Take Your Beloved Companion Away From You –There IS a Way to Save your Dog

Subtle, thoughtful argument, eh?

Steven Eisen is a “Holistic Dog Cancer Expert.” How did he get to be an expert? Well, according to his website first he read about holistic medicine in college, then he studied at the New Age Esalen Institute, and then he graduated from a college of chiropractic. Oh, and when his own dog got cancer he ignored his veterinarian’s advice and made up a bunch of stuff about diet and supplements, and since his dog lived longer than he says the vet predicted, he must have discovered the cure for dog cancer. Now, he would like you to buy the secret to curing cancer from him in the form of his book, Dog Cancer: The Holistic Answer.

I wasn’t willing to support the promotion of this nonsense by buying the book, but Dr. Eisen does offer some glimpses of his ideas on his web site and in a number of free videos. [except, of course, he will threaten to sue you if you tell anyone what he says in them]. These exhibit nearly all the warning signs of quackery, and Dr. Eisen doesn’t provide any evidence beyond his own opinion and the usual unreliable miracle stories for his claim that he has a simple, safe, effective therapy that scientific medicine either doesn’t know about or doesn’t want you to know about.

His primary web site, DogCancerAdvice.com, offers the usual sorts of unsubstantiated attacks on the ignorance and futility of conventional veterinary medicine. For example:

“The training of these professionals [veterinarians] has superbly equipped them to treat acute problems like infections, heartworm, and trauma. However, there are no standard veterinary protocols for dog cancer. The only treatments taught in veterinary school are the toxic ones…despite the scientific evidence that they are not effective. “

“Although some veterinarians claim to combine conventional and alternative dog cancer treatments, it’s rare to find one that has a thorough understanding of the principles of holistic medicine and holistic treatments for cancer in dogs. An issue that’s consistently ignored is that the two approaches are often incompatible because the underlying assumptions are worlds apart. If a gardener said he would take care of your lawn with both pesticides and organic methods, I think you would wonder about his expertise, his intelligence and/or his honesty.”

“If there were a brush fire in the forest, would it make sense to set the rest of the forest on fire? That is essentially the conventional approach to treating dog cancer.”

“Unfortunately, many have been following dangerously mistaken advice from their vets. The quality of your dog’s life can improve, and even be healed from cancer with the right information. “

“In vet speak, a dog with lymphoma is considered “cured” by conventional treatment if it survives 8 months.” [This is simply untrue. No veterinarian wuth any knowledge of cancer therapy would claim to be able to cure lymphoma. The word he’s looking for is “remission,” and the mean remission time for lymphoma with conventional treatment is about 10-14 months]

Dr. Eisen then gives detailed numbers showing so-called “holistic” treatment is cheaper than conventional cancer therapy, which seems a transparent attempt to prey on people who would like the best care possible for their pets but have significant financial limitations. He lists the potential side effects of conventional treatment without any information about how common or severe these might be and no mention of the benefits of treatment. And he finishes with the usual nonsense that Quote removed which basically says alternative cancer treatments work without any side effects. The bottom line:

“Not only are alternative treatments less toxic and more effective, they also cost just a fraction of what you would pay for conventional treatments. “

So a cheap, simple, effective cure for cancer with no risks or side effects which, for reasons we can only guess at, almost none of the veterinarians (or physicians, for that matter) know about or use. Hmmm, how likely does that seem to you? Well, let’s see what Dr. Eisen’s argument is.

1.      The Toxins Did It!

Part of being an expert in dog cancer is apparently being a toxicologist. Well, ok he’s not actually a toxicologist, but he says dog cancer is all about toxins, so we have to believe him, right?

Most of Dr. Eisen’s rant about toxins as a cause of cancer is indistinguishable from the usual oversimplified, mythologized nonsense about how the whole world is full of poisonous chemicals. The fact that some substances, both natural and human made, can cause disease, including cancer, in some people at some dose has nothing to do with the almost religious notions about contamination and purification behind the notion of “detoxification” as a treatment for disease.

Dr. Eisen lists a myriad of substances that are to blame for your pet’s disease, including vaccines, commercial pet food ingredients (including pretty much every part of the animal except muscle meat), bacteria as well as the antibiotics used to treat them, vaccines, flea and tick control products, and both tap water and bottled water (which are allegedly full of pharmaceuticals and that classic bugbear of quacks—fluoride). All of the accusations are presented as fact despite the absence of real data to support them. Nothing sells like fear!

Examples of Dr. Eisen’s sophisticated, thoughtful reasoning balancing risks and benefits include these remarks:

“If your dog has cancer, it is imperative that they never receive another vaccination. Not ever again for the rest of their life.”

“I can’t emphasize this too much. You need to eliminate the application of all commercial flea and tick products if you want to keep your dog healthy.”

2. My Diet is the Best!

Apparently, despite having no formal training in veterinary nutrition, Dr. Eisen is also an expert in cancer nutrition as well. With his usual abundance of confidence, and lack of evidence, he makes a number of claims about cancer and diet.

First, he recommends a homemade diet because, quote removed due to lega coplaint which basically says no commercial food is good for dogs with cancer. He dismisses the one cancer-specific diet on the market because of the usual myth about “byproducts” (the parts of the animal affluent Americans find yucky but that are routinely eaten by people in other cultures and universally enjoyed by wild and feral carnivores). And he completely ignores the strong evidence that homemade diets are almost always nutritionally incomplete.

Next he recommends a low-carbohydrate diet for dogs with cancer. There is some research which suggests carbohydrate restriction could have benefits for cancer patients. There is, as yet, no consistent body of evidence in humans or in dogs to indicate they actually do have benefits, so the idea is reasonable but still tentative.

He makes some arbitrary restrictions, such as banning pork as a protein source, and recommends a host of herbs which rumor, but no convincing scientific research, suggests might have anti-cancer properties. Despite the lack of any evidence to show this diet is nutritionally adequate, much less that it has a significant impact on the health or longevity of cancer patients, Dr. Eisen concludes, “This truly is healing food for your dog with cancer.”

Why? Apparently because he says so.

3.     #1 Worst Thing

Dr. Eisen states repeatedly, that cancer is growing more and more common in dogs. He bases this on reports indicating cancer is diagnosed more frequently now than ten or twenty years ago. This is likely true, though the data is not from very high quality controlled studies. Unfortunately, one area in which Dr. Eisen is not an expert is epidemiology, the study of the occurrence and causes of disease. He assumes simplistically that more cancer diagnosis means more cancer, which he then goes on to attribute, again based only on opinion, to a more toxic environment. However, more diagnosis of cancer does NOT mean that our dogs are sicker than they once were or that the environment is less healthy. Here are other reasons why cancer diagnoses can increase:

a. An Aging Population- Cancer occurs more frequently in older animals because it is frequently caused by defects in the control of cell division that accumulate over the lifetimes of many cells. The longer you live, the more times your cells have had to divide, and the more opportunities there are for the kinds of errors that can lead to cancer. Every epidemiologic study must account for differences in age between populations when comparing the rates of cancer and other disease. So if our pets are living longer as a result of better nutrition, disease prevention, and medical care, then we will see more dogs with cancer.

b. Better Control of Non-Cancerous Disease- For most of human history, almost half of all people born died before they reached adulthood. Very, very few of these children died of cancer, which is much more common in the elderly. They died mostly of malnutrition, infectious and parasitic diseases, and accidents. As science improved the quality of our food supply, provided the technology that raised our standard of living, and generated vaccines, antibiotics, pediatric and neonatal intensive care units, and all the other healthcare interventions that prevent these kinds of deaths, guess what happened? Yup, more people lived longer, the average age of the population increased, and cancer became a more common cause of death. NOT because the environment got more toxic but because it got healthier and safer! Cancer is one of the things you die of if you succeed in avoiding the age-old standards that used to kill most people.

The same is true for our pets. The very vaccines and pet foods and medicines Dr. Eisen condemns so blithely are responsible for reducing deaths from many non-cancer causes, leading to an increase in the number of cancers seen. This is a sign of past success for scientific medicine, and a challenge for the future. It is not a sign of the failure of conventional healthcare or the poisonous nature of the environment.

c. Better Diagnosis and Treatment- Diagnostic methods like ultrasound, CT scanners, and biopsies are becoming more accurate and more available all the time, and more people are willing to pay for tests to identify cancer in their dogs. This alone can increase the number of cancers diagnosed. And as we develop the ability to diagnose cancer more accurately and at an earlier stage, and as we find treatments for cancer which can (contrary to Dr. Eisen’s misleading claims) benefit cancer patients and even sometimes cure cancer, more people are willing to test their pets for these diseases and consider treating them.

I certainly wouldn’t claim that there is clear evidence cancer rates are not truly increasing. The reality is that we don’t have the appropriate epidemiological evidence to know. What is clear, however, is that the claim that there is more cancer due to increasingly toxic environments is absolutely without legitimate scientific support

So what does Dr. Esien say is the #1 Worst Thing for dogs with cancer? It’s not the commercial pet foods, flea and tick products, and tap water that he claims are contributing to a supposed cancer epidemic in dogs. Bad as these are, apparently they pale before the worst toxin of all–vaccines.

“The #1 worst thing you can do to a pet with cancer is to have them vaccinated.”

“Vaccines are loaded with toxic heavy metals like mercury and aluminum”

“Cancer is an auto-immune disease, so vaccines can cause cancer”

“Most vets administer vaccines routinely to our pets and tell you that they are safe, with very little risk. The fact is, nothing could be further from the truth.”

“Immunity is an all or nothing thing, either you have it or you don’t.”

“Any vaccination given to your dog will further impair their immune system and greatly reduce their likelihood of recovery.”

Now, there is nothing wrong with suggesting that vaccination may not be appropriate for dogs with cancer, or any other serious illness. It is widely accepted and routine to defer vaccination of dogs when they are ill. The problem with Dr. Esien’s diatribe about vaccination is that is takes this sensible recommendation and expands it into an indictment of vaccines as toxic and as a routine cause of cancer. It is like saying that because you shouldn’t drive while drunk, having a glass of wine with dinner is a major cause of auto accidents.

There are examples of vaccines being associated with an increased risk of some cancers. A type of cancer called a sarcoma can occur in between 1 out of 1,000 to 1 out of 10,000 cats vaccinated with an aluminum-containing rabies or feline leukemia vaccine. Such cancer is likely triggered by inflammation, which can also be induced by trauma, parasites, and many other causes. As a result of careful research identifying this problem (which we would not have found out about simply by casual observation or guessing), vaccines and vaccination practices have been altered to reduce this risk.

This example is often misused to suggest that vaccines in general are a risk factor for cancer in general, which is not the case. And such arguments ignore the fact that vaccinations save many, many lives. It is easier to sell an idea that is simplistic and makes the world seem like a place where all risks can be controlled and we can have the outcome we want so long as we do everything “right.” But the world we live in is more complicated than that, and anyone who tells you differently is selling something.

Dr. Eisen also belabors the fact that vaccine manufacturers state their vaccines should only be used in healthy animals. While I agree this is usually true, vaccine manufacturers also state their vaccines are safe and effective. Why does Dr. Eisen reject these claims and yet accept the recommendation not to give vaccines to ill dogs? Simply because it fits his personal beliefs about vaccination and cancer. He picks and chooses which information to accept and which to ignore not based on the strength of any evidence but on his personal prejudices. This is called confirmation bias.

Finally, Dr. Eisen also argues that vaccines aren’t effective in dogs with cancer based on manufacturer cautions about using them in immunocompromised individuals. This is a reasonable hypothesis, but one that has rarely been tested. The only specific study of the subject in dogs actually found no difference between the response to vaccination in dogs receiving chemotherapy and dogs not being treated for cancer.

Effects of chemotherapy on immune responses in dogs with cancer J Vet Intern Med. 2006 Mar-Apr;20(2):342-7. Claudia U Walter, Barbara J Biller, Susan E Lana, Annette M Bachand, Steven W Dow Animal Cancer Center, Department of Clinical Sciences, Colorado State University, Fort Collins 80523, USA.

Abstract- Chemotherapy is assumed to be immunosuppressive; yet to the authors’ knowledge, the effects of common chemotherapy protocols on adaptive immune responses in dogs with cancer have not been fully evaluated. Therefore, a study was conducted to evaluate the effects of 2 common chemotherapy protocols on T- and B-cell numbers and humoral immune responses to de novo vaccination in dogs with cancer. Twenty-one dogs with cancer (12 with lymphoma, 9 with osteosarcoma) were enrolled in a prospective study to assess effects of doxorubicin versus multi-drug chemotherapy on adaptive immunity. Numbers of circulating T and B cells were assessed by flow cytometry, and antibody responses to de novo vaccination were assessed before, during, and after chemotherapy. The T- and B-cell numbers before treatment also were compared with those of healthy, age-matched, control dogs. Prior to treatment, dogs with cancer had significantly fewer (P < .05) CD4+ T cells and CD8+ T cells than did healthy dogs. Doxorubicin treatment did not cause a significant decrease in T- or B-cell numbers, whereas treatment with combination chemotherapy caused a significant and persistent decrease in B-cell numbers. Antibody titers after vaccination were not significantly different between control and chemotherapy-treated dogs. These findings suggest that chemotherapy may have less impact on T-cell numbers and ability to mount antibody responses in dogs with cancer than was previously anticipated, though dogs with lymphoma or osteosarcoma appear to be relatively T-cell deficient before initiation of chemotherapy.

So while there are good arguments for avoiding most vaccination of dogs with cancer, Dr. Eisen isn’t making them. He is simply asserting that vaccinating dogs with cancer is ineffective (which there is evidence may not be true) and harmful (for which there is no evidence at all). In short, he is making stuff up to suit his phobia about vaccines and toxins, and he is willing to borrow and misinterpret any information from any source to do make his case.

4.     Why Buy His Book?

“What I do is help people learn about holistic protocols for treating dog cancer so they can get their dogs better without the side effects and often life-threatening consequences of conventional treatments.”

Dr. Eisen makes the usual misleading arguments for why you should take his advice. He claims his dog lived longer than expected after being diagnosed with cancer, so that proves his treatment works. Such miracle testimonials are not, however, a reliable guide to what works and what doesn’t, for reasons I’ve discussed before. He talks about individualizing therapy, which is a common marketing ploy for alternative medicine that doesn’t mean what he wants you to think it means. And he claims that many of his recommendations are “scientifically proven,” which is another empty marketing phrase. And, with the usual ambivalence towards science that characterizes most promotion of pseudoscience, he claims it as a source of legitimacy while simultaneously deriding it as worthless:

“Limiting your treatment options for dog cancer only to those proven by scientific research could be harmful to your dog’s health.” [he then goes on to talk about how individual observations are often so obvious that clinical research isn’t necessary, which is only rarely true, and how “today’s profit-driven scientific community” isn’t interested in studying treatments which are inexpensive, which is an unjustified slur on veterinarians who are just as committed to the welfare of their patients as Dr. Eisen].

Dr. Eisen offers tidbits of advice on three web sites, dogcanceradvice(dot)com (which mostly serves to advertise the book), dogcanceracademy(dot)com (which has a couple of videos you can watch in exchange for your email address, which allows more direct marketing), and bestdogcures(dot)com, which is a subscription site open to those who buy the book. All of this is classic internet marketing, very similar to what Dr. Andrew Jones does for his Veterinary Secret book and site.

Bottom Line
Dr. Eisen is a chiropractor who claims to be an expert in veterinary nutrition, toxicology, and cancer therapy based on having read a lot of books and invented his own “holistic system” by selecting which information he wanted to believe and which he felt he could ignore. The core of his marketing strategy is 1) to generate fear, by claiming that cancer is an epidemic and that we are surrounded by cancer-causing toxins in the very food and medicine we trust to keep our dogs healthy and 2) to claim that veterinarians have only dangerous and ineffective cancer treatments to offer and are ignorant of, or deliberately ignore, proven safe and effective alternative treatments. He provides no evidence for any of these claims, only his opinion and some testimonials.

A few of his claims are distantly based in real science and then stretched and twisted to support a pseudoscientific approach. Most of his claims are completely made up and contradict well-established science. He gets away with this by offering the usual unjustifiable promise of something for nothing, of effective treatments, or even cures, for cancer that are easy to use, completely safe, and cost little.

His web sites manifest many of the classic warning signs of medical quackery, including:

Claims of miraculous results
Claims of special knowledge unknown to or ignored by mainstream medicine
Claims of scientific legitimacy without any actual research data
Claims of expertise with no relevant credentials
Claims of simple and easy solutions to complex, difficult problems
Reliance on testimonials as proof
Claims that disagreement is driven by ignorance, fear, or greed
Misleading and selective use of real facts mixed in with made-up claims

And I have little doubt that, once his followers see this article, these signs will be joined by the inevitable personal attack on the critic rather than substantive response to the criticism, lots or testimonials supporting Dr. Eisen’s approach, lots of claims about the fear, anger, ignorance, greed or, most ironic of all, the arrogance of anyone who dares to criticize someone who has declared themselves an expert and who claims to know better than everyone with actual training and expertise in the subjects he talks about (true followers rarely see the irony in this).

The saddest thing about this sort of nonsense is that it is far more accessible and palatable to people whose pets have cancer than legitimate scientific information from true experts in veterinary cancer treatment. The bulk of what pet owners find on the internet is marketing materials designed to feed on their hopes and fears while not honestly helping their pets. Compassionate care is not limited to bogus “holistic protocols” like Dr. Eisen’s, despite the fact that he markets himself more effectively than genuine experts. True compassionate care is treatment based on legitimate science, not made up out of myths, misinterpretations, and wishful thinking.

Here are some real experts with real information for you if your pet has cancer:

Veterinary Oncolink, University of Pennsylvania
Pet Owner’s Guide to Cancer, Cornell University
The Cancer Center at Cares Canine Cancer Library, National Canine Cancer Foundation Animal Cancer Center, Colorado State University

Any time there is discussion of alternative medicine, of treatments that haven’t passed the test of rigorous scientific investigation (or even been tested at all), there are miracle stories. Testimonials, anecdotes, whatever you call them, they are tales of amazing recoveries from illness when everything tried had failed and all hope was lost. The comments on this blog contain many such stories for almost all of the products and methods I have questioned.

And such stories provide emotionally compelling evidence for the effectiveness of these interventions. If an owner has tried a hundred treatments for their pet, or been told there is no hope and nothing can save the animal, and then they try one more thing and their companion recovers, surely this means that final treatment must have worked? And yet, with all these testaments to the power of these alternative methods, I still have doubts. Let’s take a look at why that is.

There are a number of arguments against believing that testimonials prove anything about the effects of a medical treatment. Most of these, of course, will have no effect on the individual experiencing the miracle. It is in the nature of human beings to find their own experiences, and stories from others, deeply compelling, and sadly facts and logical arguments have little impact on that fundamentally emotional and irrational sense of certainty. Still, making the effort to understand the limitations of anecdotes has led us to this point in time, a point in which our medicine is more effective by far than any time in human history. So there is clearly value in understanding the problem of miracle stories even if many people will still find them compelling.

Things Aren’t Always What They Seem
The first classic argument is simply that things aren’t always what they seem. There are many reasons why the fact that a patient gets better after receiving a particular medical treatment doesn’t actually tell us the treatment made the patient better. The notion that if an action preceding an outcome means the action must have caused that outcome is known as the post hoc ergo propter hoc fallacy. In medicine, there is a long list of reasons why this kind of reasoning about causation isn’t trustworthy:

1. Self-limiting Disease:

Most medical problems ultimately go away by themselves. Certainly minor ones nearly always do, and even major, life-threatening diseases like cancer resolve spontaneously despite our belief that they can’t or shouldn’t. It is natural to assume that whatever we did last, right before the patient got better, is the reason they got better. Natural, but not reliably true. 

2. Regression to the Mean:

Chronic diseases, especially mysterious ones we don’t fully understand, tend to come and go. In addition to spontaneously going away altogether, which they can even after many years, they commonly tend to get better for a while then worse for a while and back and forth in an unpredictable cycle which we are always trying to understand and predict. Some stories of miraculous results from medical treatments are simply cases in which the patient was having a particularly bad run of symptoms and then felt better after getting treated.

If we systematically followed up all these stories, we would find a fair number of people whose symptoms returned despite continuing the supposedly effective treatment. Just like we find that almost any fad diet works for some people, and yet by a year most of them have gained back the weight they lost initially. Companies and practitioners selling alternative therapies, however, don’t make a point of systematically following up on all their satisfied customers’ testimonials to see which eventually got sick again. I wonder why that is? 

3. Placebo Effect:

Everyone hates this one because it means you believe something which isn’t actually true. You use a treatment and feel better, but if it’s just a placebo and your disease isn’t actually any different, that means you must be crazy or stupid, right? No! Claiming that a perceived change is only the result of a placebo effect isn’t an insult to the patient. It is simply a fact that all human beings see want we want to see a fair portion of the time, and that sometimes we interpret what we see to mean something that it doesn’t really mean.

In the case of medical therapies, almost anything I give a client, they will perceive some improvement right after simply because we’ve done something and we know we are supposed to be looking hard for any sign of an effect, especially since we really want there to be one. Again, systematically following up all these early responses often reveals no meaningful change, but if you ask at the right time, the answer will almost always be positive and suitable for use as a testimonial.

4. Misdiagnosis:

Sometimes, the reason a disease went away after an unconventional treatment (or a conventional one, for that matter) was used is because the disease was never there to begin with. Doctors aren’t perfect, and they sometimes diagnose diseases the patient doesn’t have. Any certainly people who aren’t trained medical professionals but who diagnose themselves or their pets are quite likely wrong a lot of the time. It’s easy to cure a disease that a patient didn’t have, so one of the first questions to ask about any miracle story is what hard, objective, tangible evidence is there to support the diagnosis? If your dog had cancer and it was cured by some kind of pseudoscientific nonsense like homeopathy, there had better be extensive imaging and before and after biopsies with long-term follow up before you start claiming homeopathy actually works and all the laws of physics and chemistry are mistakes.

You Can’t Prove It Isn’t a Miracle!
Another problem with miracle stories as proof that specific medical treatments actually work is that such stories can’t ever be proven wrong. And if something can’t be proven wrong no matter what, then you have to believe each and every one equally, which as we will see is probably the biggest problem with anecdotes.

Of course, the main reason you can’t disprove a miracle tale is psychological; cognitive dissonance is such a powerful and ubiquitous feature of the human brain, that it is almost impossible to convince someone they are wrong about a strongly emotional personal experience. Just as it is a great relief and comfort to know that you saved your beloved pet by using an unconventional therapy your vet said wouldn’t work, it is equally terrifying to believe that actually the therapy had nothing to do with your pet getting better and that not only did you waste your money and put your pet at unnecessary risk, but the world is ultimately unpredictable and uncontrollable and “As flies to wanton boys, are we to the gods. They kill us for their sport.” (King Lear) People cling to their miracles largely because these stories give them hope that bad things can be avoided or controlled if we just find the right things to do or believe.

Another reason miracle stories are unfalsifiable, which makes them largely useless as a form of evidence in science, is that they consist of very limited and selective information. By definition, anecdotes presented to show a treatment works contain only information that supports the effectiveness of the treatment. It wouldn’t be a miracle if at the end of the story the disease killed the patient. But does that mean there aren’t stories about the same treatment in which there was no benefit? Or even great harm? No, of course not. Testimonials are a form of “evidence” that is positive by definition, and that necessarily excludes negative evidence no matter how much of it might actually exist.

Of course, for a true believer, it would make no difference even if one could produce a story of failure for every single story of success. Cognitive dissonance would simply move the believer to say, “Well, maybe it doesn’t work for everybody, but it sure worked for me!” But for anyone trying to make an objective decision about how likely a particular therapy is to be effective, it is worth noting that testimonials and other anecdotes leave out all the stories that show the therapy failing, so they are an inherently biased kind of evidence.

Everything Works!
Though it doesn’t always show up on the typical lists of reasons why we shouldn’t trust miracle stories, to me the most damning argument against them is simply that they can be found supporting every medical treatment ever invented. On this blog alone, you can find a testimonial in the comments for nearly every product or therapy I have questioned, despite the fact that most of them have entirely different theories and methods by which they are supposed to work. And if you look at alternative medicine in general, there is not a single practice that you can’t find lots of people with powerful, emotional stories showing positive results.

But let’s go beyond the alternative therapies popular in the US today. Sure, there are plenty of testimonials for, say, Double Helix Water. How about for other kinds of magic water?

Homeopathy (after all, it’s just water)
Lourdes Water
Electron Water
pH Miracle Water
Kangen Water
Hexagon Water
Pranic Water
Ypsilanti Water
And even just Plain Water

So does every kind of water cure disease? Maybe the problem isn’t that testimonials aren’t unreliable. Maybe we’ve just stumbled across the fact that water itself is medicine? Ah, but I wonder then if we can find the same testimonials for every other therapy ever invented? Let’s see.

Christian Spiritual Healing
Islamic Spiritual Healing
Hindu Spiritual Healing
Christian Science Healing
Ancient Greek Healing
Ancient Roman Healing
Norse Magical Healing
Shamanic Healing
Theta Energy Healing
Ritual Child Sacrifice for Healing
Crystal Healing
Laser Healing
Bloodletting Healing
Magnetic Healing
Angel Therapy
Urine Therapy

Anyway, this list could be endless even without including the most popular alternative therapies. But it seems to suggest we should ask this question: If testimonials and miracle stories exist to support every medical treatment ever invented, does that mean every treatment works, or does it mean testimonials aren’t reliable?

Testimonials are useful only to generate hypotheses, or to illustrate theories that must be demonstrated by other means. By themselves, they prove nothing. Not only are they not reliable for all the reasons I’ve discussed, but if we accept them as compelling evidence, we commit ourselves to believing in everything at once, or simply arbitrarily choosing which stories to believe and which to doubt. This does not lead us to separating truth from error.