Dental Hygiene - EBP: Evaluating the Evidence

Welcome

What is EBP?

ASK
Clinical Question

ACQUIRE
Literature Search

Literature Reviews in Dental Hygiene

Test
Your Knowledge

Glossary(CEBM)

References

Evaluating the Evidence

We have now identified current information which can answer our clinical question. The next step is to read the article and evaluate the study.

There are three basic questions that need to be answered for every type of study:

• Are the results of the study valid?
• What are the results?
• Will the results help in caring for my patient?

This tutorial will focus on the first question: are the results of the study valid? The issue of validity speaks to the "truthfulness" of the information. The validity criteria should be applied before an extensive analysis of the study data. If the study is not valid, the data may not be useful.

The evidence that supports the validity or truthfulness of the information is found primarily in the study methodology. Here is where the investigators address the issue of bias, both conscious and unconscious. Study methodologies such as randomization, blinding and accounting for all patients help insure that the study results are not overly influenced by the investigators or the patients.

Evaluating the medical literature is a complex undertaking. This session will provide you with some basic criteria and information to consider when trying to decide if the study methodology is sound. You will find that the answers to the questions of validity may not always be clearly stated in the article and that clinicians will have to make their own judgments about the importance of each question.

Once you have determined that the study methodology is valid, you must examine the results and their applicability to the patient. Clinicians may have additional concerns such as whether the study represented patients similar to his/her patients, whether the study covered the aspect of the problem that is most important to the patient, or whether the study suggested a clear and useful plan of action.

Note: The questions that we used to test the validity of the evidence are adapted from work done at McMaster University. See the References/Glossary unit: 'Users' Guides to the Medical Literature.'

For examples of evaluating search results for other types of studies, review these sections from the original version of this tutorial: Therapy (3 pages) | Diagnosis | Prognosis | Etiology/Harm

1. Did the review explicitly address a sensible question?

The systematic review should address a specific question that indicates the patient problem, the exposure and one or more outcomes. General reviews, which usually do not address specific questions, may be too broad to provide an answer to the clinical question for which you are seeking information.

2. Was the search for relevant studies detailed and exhaustive?

Researchers should conduct a thorough search of appropriate bibliographic databases. The databases and search strategies should be outlined in the overview. Researchers should also show evidence of searching for non-published evidence by contacting experts in the fields. Cited references at the end of articles should also be checked.

3. Were the primary studies of high methodological quality?

Researchers should evaluate the validity of each study included in the systematic review. The same EBM criteria used to critically appraise studies should be used to evaluate studies to be included in the systematic review. Differences in study results may be explained by differences in methodology and study design.

4. Were selection and assessments of the included studies reproducible?

More than one researcher should evaluate each study and make decisions about its validity and inclusion. Bias (systematic errors) and mistakes (random errors) can be avoided when judgment is shared. A third reviewer should be available to break a tie vote.

Were the results similar from study to study?
How similar were the point estimates?
Do confidence intervals overlap between studies?

What are the overall results of the review?
Were results weighted both quantitatively and qualitatively in summary estimates?

How precise were the results?
What is the confidence interval for the summary or cumulative effect size?

Deciphering a forest plot for a systematic review/meta-analysis

More information on reading forest plots:

Ried K. Interpreting and understanding meta-analysis graphs--a practical
guide. Aust Fam Physician. 2006 Aug;35(8):635-8. PubMed PMID: 16894442.

Greenhalgh T. Papers that summarise other papers (systematic
reviews and meta-analyses). BMJ. 1997 Sep 13;315(7109):672-5.
PubMed PMID: 9310574.

Were all patient-important outcomes considered?
Did the review omit outcomes that could change decisions?

Are any postulated subgroup effects credible?
Were subgroup differences postulated before data analysis?
Were subgroup differences consistent across studies?

What is the overall quality of the evidence?
Were prevailing study design, size, and conduct reflected in a summary of the quality of evidence?

Are the benefits worth the costs and potential risks?
Does the cumulative effect size cross a test or therapeutic threshold?

Based on:  Guyatt, G. Rennie, D. Meade, MO, Cook, DJ.  Users' Guide to Medical Literature: A Manual for Evidence-Based Clinical Practice, 2nd Edition 2008.

Revised July 2010 / Adapted February 2011