Advertisement

Counterpoint: Evidence-Based Medicine Lacks a Sound Scientific Base

  • Martin J. Tobin
    Correspondence
    Division of Pulmonary and Critical Care Medicine, Loyola University of Chicago Stritch School of Medicine, Edward J. Hines, Jr, Veterans Administration Hospital, Fifth Ave and Roosevelt Road (111N), Hines, IL 60141
    Affiliations
    Hines, IL
    Search for articles by this author
      I firmly believe clinical practice should be based on the best scientific evidence. But how do you define best evidence? Evidence-based medicine (EBM) founders say “identifying the best evidence means using epidemiologic and biostatistical ways of thinking.”
      • Davidoff F
      • Haynes B
      • Sackett D
      • et al.
      Evidence based medicine.
      Table 1lists five reasons why this approach is scientifically unsound.
      Table 1Why EBM Lacks a Sound Scientific Base
      EBM grading is detached from scientific theory
      (EBM grades homeopathy as level 1 evidence)
      Failure of the attempt of logical positivism to demarcate levels of knowledge
      (EBM founders do not explain why their system can overcome what proved insurmountable to the foremost epistemologists)
      EBM reduces the methodology of science to a single step
      (EBM asserts that avoidance of assignment bias cancels every other methodologic error)
      EBM confuses statistics for science
      (Grading of clinical-practice guidelines is decided by confidence interval and totally ignores breaches of internal validity)
      EBM is not internally consistent
      (EBM has not tested itself against own standards [an RCT]; thus, by its own standards, EBM is invalid)

      Grading

      A fundamental premise on which EBM is founded is the ability to grade the quality of research studies. The grading system (levels 1 to 5 evidence) was originally published in a CHEST Supplement (Table 2).
      • Cook DJ
      • Guyatt GH
      • Laupacis A
      • et al.
      Clinical recommendations using levels of evidence for antithrombotic agents.
      EBM grading views randomization as not just one important factor but more important than every other component of research methodology. The same concept is rephrased by Sackett et al
      • Sackett DL
      • Straus SE
      • Richardson WS
      • et al.
      : “If the study wasn't randomized, we'd suggest that you stop reading it and go on to the next article.” EBM grading is based on neither empirical investigation nor rationalist theory. The original article
      • Cook DJ
      • Guyatt GH
      • Laupacis A
      • et al.
      Clinical recommendations using levels of evidence for antithrombotic agents.
      is simply an opinion piece.
      Table 2Levels of Evidence
      Modified with permission from Cook et al.2
      Level 1: RCT or metaanalysis
      (Lower limit of confidence interval for treatment effect exceeds minimal important benefit)
      Level 2: RCT or metaanalysis
      (Lower limit of confidence interval for treatment effect overlaps with minimal important benefit)
      Level 3: Nonrandomized concurrent cohort study
      Level 4: Nonrandomized historic cohort study
      Level 5: Case series without control subjects
      * Modified with permission from Cook et al.
      • Cook DJ
      • Guyatt GH
      • Laupacis A
      • et al.
      Clinical recommendations using levels of evidence for antithrombotic agents.
      There are two reasons why EBM grading is flawed. One, the grading is detached from scientific theory.
      • Cook DJ
      • Guyatt GH
      • Laupacis A
      • et al.
      Clinical recommendations using levels of evidence for antithrombotic agents.
      Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine.
      Homeopathy uses drugs in which less than one molecule of active agent is present. Benefit with dilution beyond Avogadro number contradicts pharmacologic theory. A metaanalysis
      • Linde K
      • Clausius N
      • Ramirez G
      • et al.
      Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials.
      of 89 placebo-controlled trials revealed a combined odds of 2.45 in favor of homeopathy. EBM grades metaanalysis as level 1 evidence but completely ignores scientific theory.
      • Cook DJ
      • Guyatt GH
      • Laupacis A
      • et al.
      Clinical recommendations using levels of evidence for antithrombotic agents.
      There is nothing necessarily wrong with this particular metaanalysis, but the example illustrates how a system that grades findings of all metaanalyses as level 1 evidence
      • Cook DJ
      • Guyatt GH
      • Laupacis A
      • et al.
      Clinical recommendations using levels of evidence for antithrombotic agents.
      is inherently flawed.
      • Tobin MJ
      • Jubran A
      Meta-analysis under the spotlight: focused on a meta-analysis of ventilator weaning.
      A grading system that ranks homeopathy as sounder evidence than centuries of pharmacologic science commits the reductio ad absurdum fallacy in logic.
      Two, attempts at grading of research in other disciplines have failed. The most famous attempt was by the logical positivists.
      • Haack S
      This school contained some of the brightest minds of the early twentieth century. It dominated analytic philosophy of that period. Positivists developed a verifiability criterion, which demarcated “meaningful” from “meaningless” research statements. Popper
      • Popper K
      and others pointed out two fundamental flaws of positivism; thereafter, positivism lost all supporters.
      • Haack S
      EBM retains these two flaws: a dissociation of facts from scientific theory (homeopathy, above), and no empirical testing (see below).
      EBM founders have repeatedly revised their grading system.
      GRADE Working Group. Grading quality of evidence and strength of recommendations.
      They have, however, never provided reasons why their system is capable of overcoming the problems that proved insurmountable to the logical positivists. Given the defeat of positivism, the leading epistemologists in the world have considered all attempts to grade scientific research as fundamentally flawed.
      • Haack S
      • Popper K
      • Ziman J
      No field of inquiry, other than clinical medicine, attempts to grade science.
      EBM thinking gets even more worrisome. EBM founders say evidence can be “pregraded for validity by people with expertise in research methods.”
      • Haynes B
      What kind of evidence is it that evidence-based medicine advocates want health care providers and consumers to pay attention to?.
      Wait. Surely “pregraded” is misstated. Can you grade an article before reading it? Apparently yes. That is the inevitable conclusion of an argument premised on the belief that a randomized controlled trial (RCT) always constitutes level 1 evidence (no matter how sloppy the research). This is equivalent to judging a book by its cover.

      Requirements for Reliable Research

      Table 3lists eight examples of requirements for reliable research.
      • Ziman J
      It would be silly to rank these. If one is absent, the research is no longer reliable. Yet, EBM pregrades a study as level 1 evidence if researchers avoided assignment bias (through randomization) even if they ignored the other seven requirements.
      • Cook DJ
      • Guyatt GH
      • Laupacis A
      • et al.
      Clinical recommendations using levels of evidence for antithrombotic agents.
      A grading system premised on the belief that randomization can cancel every other methodologic error is contrary to the most elementary understanding of science.
      Table 3Requirements for Reliable Clinical Research
      Avoid assignment bias
      Minimize random error
      Minimize systematic error
      Ensure accurate taxonomy
      Ensure internal validity
      Ensure external validity
      Findings that fit within the corpus of knowledge
      Reproducibility (withstands falsification attempts)

      Guidelines

      Clinicians have been lured into accepting EBM-based, clinical-practice guidelines in the belief they place medicine on a more scientific basis.
      • Guyatt GH
      • Sackett DL
      • Sinclair JC
      • et al.
      Users' guides to the medical literature: ix. A method for grading health care recommendations; Evidence-Based Medicine Working Group.
      An example familiar to CHEST readers is the grade A recommendation made by an EBM Task Force for implementation of weaning protocols.
      • Ely EW
      • Meade MO
      • Haponik EF
      • et al.
      Mechanical ventilator weaning protocols driven by nonphysician health-care professionals: evidence-based clinical practice guidelines.
      The task force refers specifically to the study by Ely et al
      • Ely EW
      • Baker AM
      • Dunagan DP
      • et al.
      Effect on the duration of mechanical ventilation of identifying patients capable of breathing spontaneously.
      as sound evidence. But this study has flawed internal validity: intermittent mandatory ventilation was used in 76% of patients in the control arm, whereas T-tube or flow-by trials were used in 100% of patients of the intervention arm.
      How could EBM founders base a grade A recommendation on a study with flawed internal validity? Because their criteria completely ignore breaches of internal validity.
      • Cook DJ
      • Guyatt GH
      • Laupacis A
      • et al.
      Clinical recommendations using levels of evidence for antithrombotic agents.
      So what is a grade A recommendation based on? It is based on “precision of the estimated intervention effects … the narrower the confidence interval … the greater our ability to make strong recommendations.”
      • Guyatt GH
      • Sackett DL
      • Sinclair JC
      • et al.
      Users' guides to the medical literature: ix. A method for grading health care recommendations; Evidence-Based Medicine Working Group.
      (I am not making this stuff up.) Confidence interval is largely determined by sample size.
      • Tobin MJ
      • Jubran A
      Meta-analysis under the spotlight: focused on a meta-analysis of ventilator weaning.
      This type of “precision” has nothing to do with “scientific precision,” such as ensuring internal validity.
      • Tobin MJ
      • Jubran A
      Meta-analysis under the spotlight: focused on a meta-analysis of ventilator weaning.
      The graders' emphasis on confidence interval confuses statistics with science. The small confidence interval is a trap for the nonthinker: statistical precision is misinterpreted as “scientific exactness.”

      Harm

      You may think EBM does no harm. Not so. Clinical medicine requires thoughtful reflection about each individual patient, whereas graded guidelines encourage reflexive action. A double-blind RCT revealed that spironolactone decreased the mortality rate in patients with severe congestive heart failure (CHF) by 30%.
      • Pitt B
      • Zannad F
      • Remme WJ
      • et al.
      The effect of spironolactone on morbidity and mortality in patients with severe heart failure: randomized Aldactone Evaluation Study Investigators.
      The clinical practice guidelines of the American Heart Association subsequently recommended spironolactone for treatment of ventricular dysfunction.
      Report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee To Revise the 1995 Guidelines for the Evaluation and Management of Heart Failure): ACC/AHA guidelines for the evaluation and management of chronic heart failure in the adult, executive summary.
      This was followed by a fourfold increase in spironolactone prescriptions, and a sixfold increase in death from hyperkalemia.
      • Juurlink DN
      • Mamdani MM
      • Lee DS
      • et al.
      Rates of hyperkalemia after publication of the Randomized Aldactone Evaluation Study.
      Reflex response to level 1 evidence, without reflection about underlying pathophysiology and individual context, can kill.
      Guidelines based on level 1 evidence, which ignore non-RCT research, can also kill. Sinuff et al
      • Sinuff T
      • Cook DJ
      • Randall J
      • Allen CJ
      Evaluation of a practice guideline for noninvasive positive-pressure ventilation for acute respiratory failure.
      developed a guideline for use of noninvasive positive-pressure ventilation (NPPV) in acute respiratory failure. They judged RCT data to support use of NPPV in COPD and CHF but not in other conditions. Before the guideline was introduced, 35% of patients with conditions other than COPD and CHF were intubated. After the guideline came into force, 100% were intubated,
      • Sinuff T
      • Cook DJ
      • Randall J
      • Allen CJ
      Evaluation of a practice guideline for noninvasive positive-pressure ventilation for acute respiratory failure.
      and mortality increased from 21 to 34%. Hill
      • Hill NS
      Practice guidelines for noninvasive positive-pressure ventilation: help or hindrance?.
      pointed out that by classifying “patients as not meeting NPPV criteria, the authors could have unintentionally encouraged endotracheal intubation in this subgroup, possibly contributing to morbidity and mortality.”

      EBM Proves That EBM Is Unsound

      The fundamental assumption of EBM is that physicians who practice EBM provide superior care.
      Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine.
      But EBM founders have never undertaken an RCT of the effect of EBM on patient outcome.
      • Haynes B
      What kind of evidence is it that evidence-based medicine advocates want health care providers and consumers to pay attention to?.
      So EBM does not satisfy its own basic requirements, which it demands of everyone else. (Hypocrisy or what?)
      They say an RCT of EBM is unnecessary because “outcomes researchers consistently document that patients who receive proven efficacious therapies have better outcomes than those who do not.”
      • Straus SE
      • McAlister FA
      Evidence-based medicine: a commentary on common criticisms.
      With this non sequitur, EBM advocates claim credit for all research done under the heading of clinical research. But EBM is not a product of research. It is an activity for ranking the products of research. EBM advocates conflate the two. They need to disentangle them.
      EBM founders say an RCT of EBM would be “impossible to do,”
      • Haynes B
      What kind of evidence is it that evidence-based medicine advocates want health care providers and consumers to pay attention to?.
      another non sequitur. Not true. All that is needed is to undertake a matched comparison of institutions where physicians practice EBM vs institutions where physicians do not believe in the tenets of EBM.
      EBM founders say clinical decisions should be based on empirical evidence, and that expert opinion is untrustworthy.
      • Cook DJ
      • Guyatt GH
      • Laupacis A
      • et al.
      Clinical recommendations using levels of evidence for antithrombotic agents.
      • Sackett DL
      • Straus SE
      • Richardson WS
      • et al.
      Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine.
      • Haynes B
      What kind of evidence is it that evidence-based medicine advocates want health care providers and consumers to pay attention to?.
      But EBM founders have never subjected EBM to empirical testing. Instead, EBM (and grading) is solely based on expert opinion. Thus, if EBMs tenets are true, then EBM should not be trusted, quod erat demonstrandum.

      What Is the Alternative?

      A major attraction of EBM is that it offers a means of coping with uncertainty. Given a physician's responsibility—to make life-and-death decisions about another human—the wish for certainty is understandable, as is the wish of wanting to act like the wisest physician when faced with a problematic patient. But these wishes are contrary to the reality of medicine.
      A wise physician makes decisions on a background of scientific theory (universal principles) [Fig 1]. Clinical practice, however, involves primarily phronesis (practical wisdom): a customized decision for one particular patient. A wise clinician bases each customized decision on a sound knowledge of science. Many physicians have been seduced by marketing of the “EBM-grading construct,” believing it makes clinical practice more scientific. These physicians, however, seem unaware that the EBM-grading construct is detached from science and poses a serious risk to patient safety.
      Figure thumbnail gr1
      Figure 1In The School of Athens (1510–1511; Stanza della Segnatura, Vaticano), Raphael captures the rational search for truth. Plato is pointing up to the heavens, emphasizing epistemic knowledge, that of theoretical universals. His pupil, Aristotle, has his hand turned down to earth, emphasizing phronesis (practical wisdom). A wise clinician has a foot in both camps. The notion of an external repository of evidence, pregraded for validity,
      • Haynes B
      What kind of evidence is it that evidence-based medicine advocates want health care providers and consumers to pay attention to?.
      which can be applied impersonally in a logical deductive manner, is a cherished belief of EBM advocates
      • Cook DJ
      • Guyatt GH
      • Laupacis A
      • et al.
      Clinical recommendations using levels of evidence for antithrombotic agents.
      • Sackett DL
      • Straus SE
      • Richardson WS
      • et al.
      Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine.
      GRADE Working Group. Grading quality of evidence and strength of recommendations.
      • Haynes B
      What kind of evidence is it that evidence-based medicine advocates want health care providers and consumers to pay attention to?.
      • Guyatt GH
      • Sackett DL
      • Sinclair JC
      • et al.
      Users' guides to the medical literature: ix. A method for grading health care recommendations; Evidence-Based Medicine Working Group.
      • Ely EW
      • Meade MO
      • Haponik EF
      • et al.
      Mechanical ventilator weaning protocols driven by nonphysician health-care professionals: evidence-based clinical practice guidelines.
      • Sinuff T
      • Cook DJ
      • Randall J
      • Allen CJ
      Evaluation of a practice guideline for noninvasive positive-pressure ventilation for acute respiratory failure.
      and malpractice attorneys.

      References

        • Davidoff F
        • Haynes B
        • Sackett D
        • et al.
        Evidence based medicine.
        BMJ. 1995; 310: 1085-1086
        • Cook DJ
        • Guyatt GH
        • Laupacis A
        • et al.
        Clinical recommendations using levels of evidence for antithrombotic agents.
        Chest. 1995; 108: 227S-230S
        • Sackett DL
        • Straus SE
        • Richardson WS
        • et al.
        Evidence-based medicine: how to practice and teach EBM. Churchill Livingstone, Edinburgh, UK2000: 108 (2nd ed.)
      1. Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine.
        JAMA. 1992; 268: 2420-2425
        • Linde K
        • Clausius N
        • Ramirez G
        • et al.
        Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials.
        Lancet. 1997; 350: 834-843
        • Tobin MJ
        • Jubran A
        Meta-analysis under the spotlight: focused on a meta-analysis of ventilator weaning.
        Crit Care Med. 2008; 36: 1-7
        • Haack S
        Defending science–within reason: between scientism and cynicism. Prometheus Books, New York, NY2007
        • Popper K
        The logic of scientific discovery. Hutchison, London, UK1958
      2. GRADE Working Group. Grading quality of evidence and strength of recommendations.
        BMJ. 2004; 328: 1490-1494
        • Ziman J
        Real science: what it is, and what it means. Cambridge University Press, Cambridge, UK2000
        • Haynes B
        What kind of evidence is it that evidence-based medicine advocates want health care providers and consumers to pay attention to?.
        BMC Health Serv Res. 2002; 2: 1-7
        • Guyatt GH
        • Sackett DL
        • Sinclair JC
        • et al.
        Users' guides to the medical literature: ix. A method for grading health care recommendations; Evidence-Based Medicine Working Group.
        JAMA. 1995; 274: 1800-1804
        • Ely EW
        • Meade MO
        • Haponik EF
        • et al.
        Mechanical ventilator weaning protocols driven by nonphysician health-care professionals: evidence-based clinical practice guidelines.
        Chest. 2001; 120: 454S-463S
        • Ely EW
        • Baker AM
        • Dunagan DP
        • et al.
        Effect on the duration of mechanical ventilation of identifying patients capable of breathing spontaneously.
        N Engl J Med. 1996; 335: 1864-1869
        • Pitt B
        • Zannad F
        • Remme WJ
        • et al.
        The effect of spironolactone on morbidity and mortality in patients with severe heart failure: randomized Aldactone Evaluation Study Investigators.
        N Engl J Med. 1999; 341: 709-717
      3. Report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee To Revise the 1995 Guidelines for the Evaluation and Management of Heart Failure): ACC/AHA guidelines for the evaluation and management of chronic heart failure in the adult, executive summary.
        Circulation. 2001; 104: 2996-3007
        • Juurlink DN
        • Mamdani MM
        • Lee DS
        • et al.
        Rates of hyperkalemia after publication of the Randomized Aldactone Evaluation Study.
        N Engl J Med. 2004; 351: 543-551
        • Sinuff T
        • Cook DJ
        • Randall J
        • Allen CJ
        Evaluation of a practice guideline for noninvasive positive-pressure ventilation for acute respiratory failure.
        Chest. 2003; 123: 2062-2073
        • Hill NS
        Practice guidelines for noninvasive positive-pressure ventilation: help or hindrance?.
        Chest. 2003; 123: 1784-1786
        • Straus SE
        • McAlister FA
        Evidence-based medicine: a commentary on common criticisms.
        Can Med Assoc J. 2000; 163: 837-841