literature reviews and evidence based practice

Evidence-Based Practice (EBP)

• The EBP Process
• Forming a Clinical Question
• Inclusion & Exclusion Criteria
• Acquiring Evidence
• Appraising the Quality of the Evidence
• Writing a Literature Review
• Finding Psychological Tests & Assessment Instruments

What Is a Literature Review?

A literature review is an integrated analysis of scholarly writings that are related directly to your research question. Put simply, it's  a critical evaluation of what's already been written on a particular topic . It represents the literature that provides background information on your topic and shows a connection between those writings and your research question.

A literature review may be a stand-alone work or the introduction to a larger research paper, depending on the assignment. Rely heavily on the guidelines your instructor has given you.

What a Literature Review Is Not:

• A list or summary of sources
• An annotated bibliography
• A grouping of broad, unrelated sources
• A compilation of everything that has been written on a particular topic
• Literary criticism (think English) or a book review

Why Literature Reviews Are Important

• They explain the background of research on a topic
• They demonstrate why a topic is significant to a subject area
• They discover relationships between research studies/ideas
• They identify major themes, concepts, and researchers on a topic
• They identify critical gaps and points of disagreement
• They discuss further research questions that logically come out of the previous studies

To Learn More about Conducting and Writing a Lit Review . . .

Monash University (in Australia) has created several extremely helpful, interactive tutorials. 

• The Stand-Alone Literature Review, https://www.monash.edu/rlo/assignment-samples/science/stand-alone-literature-review
• Researching for Your Literature Review,  https://guides.lib.monash.edu/researching-for-your-literature-review/home
• Writing a Literature Review,  https://www.monash.edu/rlo/graduate-research-writing/write-the-thesis/writing-a-literature-review

Keep Track of Your Sources!

A citation manager can be helpful way to work with large numbers of citations. See UMSL Libraries' Citing Sources guide for more information. Personally, I highly recommend Zotero —it's free, easy to use, and versatile. If you need help getting started with Zotero or one of the other citation managers, please contact a librarian.

• << Previous: Appraising the Quality of the Evidence
• Next: Finding Psychological Tests & Assessment Instruments >>
• Last Updated: May 16, 2024 2:44 PM
• URL: https://libguides.umsl.edu/ebp

Masks Strongly Recommended but Not Required in Maryland, Starting Immediately

Due to the downward trend in respiratory viruses in Maryland, masking is no longer required but remains strongly recommended in Johns Hopkins Medicine clinical locations in Maryland. Read more .

• Vaccines  
• Masking Guidelines
• Visitor Guidelines  

Center for Nursing Inquiry

Evidence-based practice, what is ebp.

As nurses, we often hear the term evidence-based practice (EBP). But, what does it actually mean? EBP is a process used to review, analyze, and translate the latest scientific evidence. The goal is to quickly incorporate the best available research, along with clinical experience and patient preference, into clinical practice, so nurses can make informed patient-care decisions ( Dang et al., 2022 ). EBP is the cornerstone of clinical practice. Integrating EBP into your nursing practice improves quality of care and patient outcomes.

How do I get involved in EBP?

As a nurse, you will have plenty of opportunities to get involved in EBP. Take that “AHA” moment. Do you think there’s a better way to do something? Let’s turn to the evidence and find out!

When conducting an EBP project, it is important to use a model to help guide your work. In the Johns Hopkins Health System, we use the Johns Hopkins Evidence-Based Practice (JHEBP) model. It is a three-phase approach referred to as the PET process: practice question, evidence, and translation. In the first phase, the team develops a practice question by identifying the patient population, interventions, and outcomes (PICO). In the second phase, a literature search is performed, and the evidence is appraised for strength and quality. In the third phase, the findings are synthesized to develop recommendations for practice.

The JHEBP model is accompanied by user-friendly tools. The tools walk you through each phase of the project. Johns Hopkins nurses can access the tools via our Inquiry Toolkit . The tools are available to individuals from other institutions via the Institute for Johns Hopkins Nursing (IJHN) .

If you’re interested in learning more about the JHEBP model and tools, Johns Hopkins nurses have access to a free online course entitled JHH Nursing | Central | Evidence-Based Practice Series in MyLearning. The course follows the JHEBP process from beginning to end and provides guidance to the learner on how to use the JHEBP tools. The course is available to individuals from other institutions for a fee via the Institute for Johns Hopkins Nursing (IJHN) .

Where should I start?

All EBP projects need to be submitted to the Center for Nursing Inquiry for review. The CNI ensures all nurse-led EBP projects are high-quality and value added. We also offer expert guidance and support, if needed.

Who can help me?

The Center for Nursing Inquiry  can answer any questions you may have about the JHEBP tools. All 10 JHEBP tools can be found in our Inquiry Toolkit : project management guide, question development tool, stakeholder analysis tool, evidence level and quality guide, research evidence appraisal tool, non-research evidence appraisal tool, individual evidence summary tool, synthesis process and recommendations tool, action planning tool, and dissemination tool. The tools walk you through each phase of an EBP project.

The Welch Medical Library  serves the information needs of the faculty, staff, and students of Johns Hopkins Medicine, Nursing and Public Health. Often, one of the toughest parts of conducting an EBP project is finding the evidence. The informationist  assigned to your department can assist you with your literature search and citation management.

When do I share my work?

Your project is complete. Now what? It’s time to share your project with the scholarly community.

To prepare your EBP project for publication, use the JHEBP Dissemination Tool . The JHEBP Dissemination Tool (Appendix J) details what to include in each section of your manuscript, from the introduction to the discussion, and shows you which EBP appendices correspond to each part of a scientific paper. You can find the JHEBP Dissemination Tool in our Inquiry Toolkit . 

You can also present your project at a local, regional, or national conference. Poster and podium presentation templates are available in our Inquiry Toolkit .

To learn more about sharing your project, check out our Abstract & Manuscript Writing webinar and our Poster & Podium Presentations webinar !

Submit Your Project

Do you have an idea for an EBP project?

University Libraries

• Ohio University Libraries
• Library Guides

Evidence-based Practice in Healthcare

• Performing a Literature Review
• EBP Tutorials
• Question- PICO
• Definitions
• Systematic Reviews
• Levels of Evidence
• Finding Evidence
• Filter by Study Type
• Too Much or Too Little?
• Critical Appraisal
• Quality Improvement (QI)
• Contact - Need Help?

Hanna's Performing a qualitity literature review presentation slides

• Link to the PPT slides via OneDrive anyone can view

Characteristics of a Good Literature Review in Health & Medicine

Clear Objectives and Research Questions : The review should start with clearly defined objectives and research questions that guide the scope and focus of the review.

Comprehensive Coverage : Include a wide range of relevant sources, such as research articles, review papers, clinical guidelines, and books. Aim for a broad understanding of the topic, covering historical developments and current advancements. To do this, an intentional and minimally biased search strategy.

• Link to relevant databases to consider for a comprehensive search (search 2+ databases)
• Link to the video "Searching your Topic: Strategies and Efficiencies" by Hanna Schmillen
• Link to the worksheet "From topic, to PICO, to search strategy" to help researchers work through their topic into an intentional search strategy by Hanna Schmillen

Transparency and Replicability : The review process, search strategy, should be transparent, with detailed documentation of all steps taken. This allows others to replicate the review or update it in the future.

Appraisal of Studies Included : Each included study should be critically appraised for methodological quality and relevance. Use standardized appraisal tools to assess the risk of bias and the quality of evidence.

• Link to the video " Evaluating Health Research" by Hanna Schmillen
• Link to evaluating and appraising studies tab, which includes a rubric and checklists

Clear Synthesis and Discussion of Findings : The review should provide a thorough discussion of the findings, including any patterns, relationships, or trends identified in the literature. Address the strengths and limitations of the reviewed studies and the review itself. Present findings in a balanced and unbiased manner, avoiding over interpretation or selective reporting of results.

Implications for Practice and Research : The review should highlight the practical implications of the findings for medical practice and policy. It should also identify gaps in the current literature and suggest areas for future research.

Referencing and Citation : Use proper citation practices to credit original sources. Provide a comprehensive reference list to guide readers to the original studies.

• Link to Citation Style Guide, includes tab about Zotero

Note: A literature review is not a systematic review. For more information about systematic reviews and different types of evidence synthesis projects, see the Evidence Synthesis guide .

• << Previous: Quality Improvement (QI)
• Next: Contact - Need Help? >>

Evidence-Based Practice: Literature Reviews / Systematic Reviews

• What is EBP?
• PICO and SPIDER
• More Resources
• Types of Research
• Levels of Evidence
• How to Search for Evidence
• Where to Search for Evidence
• Literature Reviews / Systematic Reviews
• Clinical Colleagues as a Source of Evidence
• What is Being Appraised?
• Critical Appraisal Tools
• Step 4: Apply
• Step 5: Assess/Audit
• EBP Resources by Discipline

Introduction

• is based on database searches
• summarises the results of research
• has the aim of objectively discussing a specific topic or theme.

There are many types of literature review, but two of the main ones are:

• narrative (or traditional) review
• systematic review.

The image above describes common review types in terms of speed, detail, risk of bias, and comprehensiveness.

For more information on all types of literature reviews , see the Library's  Literature Review  guide. This guide includes a wealth of information on all types of reviews.

For more information on systematic reviews , see the Library's Systematic and systematic-like reviews guide.

[ Image attribution :  "Schematic of the main differences between the types of literature review" by Brennan, M. L., Arlt, S. P., Belshaw, Z., Buckley, L., Corah, L., Doit, H., Fajt, V. R., Grindlay, D., Moberly, H. K., Morrow, L. D., Stavisky, J., & White, C. (2020). Critically Appraised Topics (CATs) in veterinary medicine: Applying evidence in clinical practice. Frontiers in Veterinary Science, 7 , 314. https://doi.org/10.3389/fvets.2020.00314 is licensed under CC BY 3.0 .]

• << Previous: Where to Search for Evidence
• Next: Clinical Colleagues as a Source of Evidence >>
• Last Updated: Jun 21, 2024 9:42 AM
• URL: https://libguides.csu.edu.au/ebp

Charles Sturt University is an Australian University, TEQSA Provider Identification: PRV12018. CRICOS Provider: 00005F.

•   [email protected]

NUR 506 - Evidence Based Practice

Need more help, ask a librarian.

   [email protected]

SNHU Connect

   Nursing & Healthcare Learning Community

What is a literature review?

Part of your final project is to conduct a literature review that shows a progressive development of ideas and explains the current state of the research surrounding your PICO question and rationale for the project. Use the resources below to learn more about how to approach the literature review and best practices.

An Introduction to Literature Reviews

• Article: Literature Review in Encyclopedia of Evaluation A literature review is both process and product. The literature review process entails a systematic examination of prior research, evaluation studies, and scholarship to answer questions of theory, policy, and practice. Read this short article entry from the Encyclopedia of Evaluation to learn about literature reviews and how an integrative review is a specific type.
• eBook Chapter: Literature Review in Nursing Research and Statistics Chapter 5 of Nursing Research and Statistics, detailing the concept of the literature review, its importance, purpose, the different types etc.
• << Previous: Peer Reviewed Sources
• Next: Theoretical Framework >>

JAY SIWEK, M.D., MARGARET L. GOURLAY, M.D., DAVID C. SLAWSON, M.D., AND ALLEN F. SHAUGHNESSY, PHARM.D.

Am Fam Physician. 2002;65(2):251-258

Traditional clinical review articles, also known as updates, differ from systematic reviews and meta-analyses. Updates selectively review the medical literature while discussing a topic broadly. Nonquantitative systematic reviews comprehensively examine the medical literature, seeking to identify and synthesize all relevant information to formulate the best approach to diagnosis or treatment. Meta-analyses (quantitative systematic reviews) seek to answer a focused clinical question, using rigorous statistical analysis of pooled research studies. This article presents guidelines for writing an evidence-based clinical review article for American Family Physician . First, the topic should be of common interest and relevance to family practice. Include a table of the continuing medical education objectives of the review. State how the literature search was done and include several sources of evidence-based reviews, such as the Cochrane Collaboration, BMJ's Clinical Evidence , or the InfoRetriever Web site. Where possible, use evidence based on clinical outcomes relating to morbidity, mortality, or quality of life, and studies of primary care populations. In articles submitted to American Family Physician , rate the level of evidence for key recommendations according to the following scale: level A (randomized controlled trial [RCT], meta-analysis); level B (other evidence); level C (consensus/expert opinion). Finally, provide a table of key summary points.

American Family Physician is particularly interested in receiving clinical review articles that follow an evidence-based format. Clinical review articles, also known as updates, differ from systematic reviews and meta-analyses in important ways. 1 Updates selectively review the medical literature while discussing a topic broadly. An example of such a topic is, “The diagnosis and treatment of myocardial ischemia.” Systematic reviews comprehensively examine the medical literature, seeking to identify and synthesize all relevant information to formulate the best approach to diagnosis or treatment. Examples are many of the systematic reviews of the Cochrane Collaboration or BMJ's Clinical Evidence compendium. Meta-analyses are a special type of systematic review. They use quantitative methods to analyze the literature and seek to answer a focused clinical question, using rigorous statistical analysis of pooled research studies. An example is, “Do beta blockers reduce mortality following myocardial infarction?”

The best clinical review articles base the discussion on existing systematic reviews and meta-analyses, and incorporate all relevant research findings about the management of a given disorder. Such evidence-based updates provide readers with powerful summaries and sound clinical guidance.

In this article, we present guidelines for writing an evidence-based clinical review article, especially one designed for continuing medical education (CME) and incorporating CME objectives into its format. This article may be read as a companion piece to a previous article and accompanying editorial about reading and evaluating clinical review articles. 1 , 2 Some articles may not be appropriate for an evidence-based format because of the nature of the topic, the slant of the article, a lack of sufficient supporting evidence, or other factors. We encourage authors to review the literature and, wherever possible, rate key points of evidence. This process will help emphasize the summary points of the article and strengthen its teaching value.

Topic Selection

Choose a common clinical problem and avoid topics that are rarities or unusual manifestations of disease or that have curiosity value only. Whenever possible, choose common problems for which there is new information about diagnosis or treatment. Emphasize new information that, if valid, should prompt a change in clinical practice, such as the recent evidence that spironolactone therapy improves survival in patients who have severe congestive heart failure. 3 Similarly, new evidence showing that a standard treatment is no longer helpful, but may be harmful, would also be important to report. For example, patching most traumatic corneal abrasions may actually cause more symptoms and delay healing compared with no patching. 4

Searching the Literature

When searching the literature on your topic, please consult several sources of evidence-based reviews ( Table 1 ) . Look for pertinent guidelines on the diagnosis, treatment, or prevention of the disorder being discussed. Incorporate all high-quality recommendations that are relevant to the topic. When reviewing the first draft, look for all key recommendations about diagnosis and, especially, treatment. Try to ensure that all recommendations are based on the highest level of evidence available. If you are not sure about the source or strength of the recommendation, return to the literature, seeking out the basis for the recommendation.

In particular, try to find the answer in an authoritative compendium of evidence-based reviews, or at least try to find a meta-analysis or well-designed randomized controlled trial (RCT) to support it. If none appears to be available, try to cite an authoritative consensus statement or clinical guideline, such as a National Institutes of Health Consensus Development Conference statement or a clinical guideline published by a major medical organization. If no strong evidence exists to support the conventional approach to managing a given clinical situation, point this out in the text, especially for key recommendations. Keep in mind that much of traditional medical practice has not yet undergone rigorous scientific study, and high-quality evidence may not exist to support conventional knowledge or practice.

Patient-Oriented vs. Disease-Oriented Evidence

With regard to types of evidence, Shaughnessy and Slawson 5 – 7 developed the concept of Patient-Oriented Evidence that Matters (POEM), in distinction to Disease-Oriented Evidence (DOE). POEM deals with outcomes of importance to patients, such as changes in morbidity, mortality, or quality of life. DOE deals with surrogate end points, such as changes in laboratory values or other measures of response. Although the results of DOE sometimes parallel the results of POEM, they do not always correspond ( Table 2 ) . 2 When possible, use POEM-type evidence rather than DOE. When DOE is the only guidance available, indicate that key clinical recommendations lack the support of outcomes evidence. Here is an example of how the latter situation might appear in the text: “Although prostate-specific antigen (PSA) testing identifies prostate cancer at an early stage, it has not yet been proved that PSA screening improves patient survival.” (Note: PSA testing is an example of DOE, a surrogate marker for the true outcomes of importance—improved survival, decreased morbidity, and improved quality of life.)

Evaluating the Literature

Evaluate the strength and validity of the literature that supports the discussion (see the following section, Levels of Evidence). Look for meta-analyses, high-quality, randomized clinical trials with important outcomes (POEM), or well-designed, nonrandomized clinical trials, clinical cohort studies, or case-controlled studies with consistent findings. In some cases, high-quality, historical, uncontrolled studies are appropriate (e.g., the evidence supporting the efficacy of Papanicolaou smear screening). Avoid anecdotal reports or repeating the hearsay of conventional wisdom, which may not stand up to the scrutiny of scientific study (e.g., prescribing prolonged bed rest for low back pain).

Look for studies that describe patient populations that are likely to be seen in primary care rather than subspecialty referral populations. Shaughnessy and Slawson's guide for writers of clinical review articles includes a section on information and validity traps to avoid. 2

Levels of Evidence

Readers need to know the strength of the evidence supporting the key clinical recommendations on diagnosis and treatment. Many different rating systems of varying complexity and clinical relevance are described in the medical literature. Recently, the third U.S. Preventive Services Task Force (USPSTF) emphasized the importance of rating not only the study type (RCT, cohort study, case-control study, etc.), but also the study quality as measured by internal validity and the quality of the entire body of evidence on a topic. 8

While it is important to appreciate these evolving concepts, we find that a simplified grading system is more useful in AFP . We have adopted the following convention, using an ABC rating scale. Criteria for high-quality studies are discussed in several sources. 8 , 9 See the AFP Web site ( www.aafp.org/afp/authors ) for additional information about levels of evidence and see the accompanying editorial in this issue discussing the potential pitfalls and limitations of any rating system.

Level A (randomized controlled trial/meta-analysis): High-quality randomized controlled trial (RCT) that considers all important outcomes. High-quality meta-analysis (quantitative systematic review) using comprehensive search strategies.

Level B (other evidence): A well-designed, nonrandomized clinical trial. A nonquantitative systematic review with appropriate search strategies and well-substantiated conclusions. Includes lower quality RCTs, clinical cohort studies, and case-controlled studies with non-biased selection of study participants and consistent findings. Other evidence, such as high-quality, historical, uncontrolled studies, or well-designed epidemiologic studies with compelling findings, is also included.

Level C (consensus/expert opinion): Consensus viewpoint or expert opinion.

Each rating is applied to a single reference in the article, not to the entire body of evidence that exists on a topic. Each label should include the letter rating (A, B, C), followed by the specific type of study for that reference. For example, following a level B rating, include one of these descriptors: (1) nonrandomized clinical trial; (2) nonquantitative systematic review; (3) lower quality RCT; (4) clinical cohort study; (5) case-controlled study; (6) historical uncontrolled study; (7) epidemiologic study.

Here are some examples of the way evidence ratings should appear in the text:

“To improve morbidity and mortality, most patients in congestive heart failure should be treated with an angiotensin-converting enzyme inhibitor. [Evidence level A, RCT]”

“The USPSTF recommends that clinicians routinely screen asymptomatic pregnant women 25 years and younger for chlamydial infection. [Evidence level B, non-randomized clinical trial]”

“The American Diabetes Association recommends screening for diabetes every three years in all patients at high risk of the disease, including all adults 45 years and older. [Evidence level C, expert opinion]”

When scientifically strong evidence does not exist to support a given clinical recommendation, you can point this out in the following way:

“Physical therapy is traditionally prescribed for the treatment of adhesive capsulitis (frozen shoulder), although there are no randomized outcomes studies of this approach.”

Format of the Review

Introduction.

The introduction should define the topic and purpose of the review and describe its relevance to family practice. The traditional way of doing this is to discuss the epidemiology of the condition, stating how many people have it at one point in time (prevalence) or what percentage of the population is expected to develop it over a given period of time (incidence). A more engaging way of doing this is to indicate how often a typical family physician is likely to encounter this problem during a week, month, year, or career. Emphasize the key CME objectives of the review and summarize them in a separate table entitled “CME Objectives.”

The methods section should briefly indicate how the literature search was conducted and what major sources of evidence were used. Ideally, indicate what predetermined criteria were used to include or exclude studies (e.g., studies had to be independently rated as being high quality by an established evaluation process, such as the Cochrane Collaboration). Be comprehensive in trying to identify all major relevant research. Critically evaluate the quality of research reviewed. Avoid selective referencing of only information that supports your conclusions. If there is controversy on a topic, address the full scope of the controversy.

The discussion can then follow the typical format of a clinical review article. It should touch on one or more of the following subtopics: etiology, pathophysiology, clinical presentation (signs and symptoms), diagnostic evaluation (history, physical examination, laboratory evaluation, and diagnostic imaging), differential diagnosis, treatment (goals, medical/surgical therapy, laboratory testing, patient education, and follow-up), prognosis, prevention, and future directions.

The review will be comprehensive and balanced if it acknowledges controversies, unresolved questions, recent developments, other viewpoints, and any apparent conflicts of interest or instances of bias that might affect the strength of the evidence presented. Emphasize an evidence-supported approach or, where little evidence exists, a consensus viewpoint. In the absence of a consensus viewpoint, you may describe generally accepted practices or discuss one or more reasoned approaches, but acknowledge that solid support for these recommendations is lacking.

In some cases, cost-effectiveness analyses may be important in deciding how to implement health care services, especially preventive services. 10 When relevant, mention high-quality cost-effectiveness analyses to help clarify the costs and health benefits associated with alternative interventions to achieve a given health outcome. Highlight key points about diagnosis and treatment in the discussion and include a summary table of the key take-home points. These points are not necessarily the same as the key recommendations, whose level of evidence is rated, although some of them will be.

Use tables, figures, and illustrations to highlight key points, and present a step-wise, algorithmic approach to diagnosis or treatment when possible.

Rate the evidence for key statements, especially treatment recommendations. We expect that most articles will have at most two to four key statements; some will have none. Rate only those statements that have corresponding references and base the rating on the quality and level of evidence presented in the supporting citations. Use primary sources (original research, RCTs, meta-analyses, and systematic reviews) as the basis for determining the level of evidence. In other words, the supporting citation should be a primary research source of the information, not a secondary source (such as a nonsystematic review article or a textbook) that simply cites the original source. Systematic reviews that analyze multiple RCTs are good sources for determining ratings of evidence.

The references should include the most current and important sources of support for key statements (i.e., studies referred to, new information, controversial material, specific quantitative data, and information that would not usually be found in most general reference textbooks). Generally, these references will be key evidence-based recommendations, meta-analyses, or landmark articles. Although some journals publish exhaustive lists of reference citations, AFP prefers to include a succinct list of key references. (We will make more extensive reference lists available on our Web site or provide links to your personal reference list.)

You may use the following checklist to ensure the completeness of your evidence-based review article; use the source list of reviews to identify important sources of evidence-based medicine materials.

Checklist for an Evidence-Based Clinical Review Article

The topic is common in family practice, especially topics in which there is new, important information about diagnosis or treatment.

The introduction defines the topic and the purpose of the review, and describes its relevance to family practice.

A table of CME objectives for the review is included.

The review states how you did your literature search and indicates what sources you checked to ensure a comprehensive assessment of relevant studies (e.g., MEDLINE, the Cochrane Collaboration Database, the Center for Research Support, TRIP Database).

Several sources of evidence-based reviews on the topic are evaluated ( Table 1 ) .

Where possible, POEM (dealing with changes in morbidity, mortality, or quality of life) rather than DOE (dealing with mechanistic explanations or surrogate end points, such as changes in laboratory tests) is used to support key clinical recommendations ( Table 2 ) .

Studies of patients likely to be representative of those in primary care practices, rather than subspecialty referral centers, are emphasized.

Studies that are not only statistically significant but also clinically significant are emphasized; e.g., interventions with meaningful changes in absolute risk reduction and low numbers needed to treat. (See http://www.cebm.net/index.aspx?o=1116 .) 11

The level of evidence for key clinical recommendations is labeled using the following rating scale: level A (RCT/meta-analysis), level B (other evidence), and level C (consensus/expert opinion).

Acknowledge controversies, recent developments, other viewpoints, and any apparent conflicts of interest or instances of bias that might affect the strength of the evidence presented.

Highlight key points about diagnosis and treatment in the discussion and include a summary table of key take-home points.

Use tables, figures, and illustrations to highlight key points and present a step-wise, algorithmic approach to diagnosis or treatment when possible.

Emphasize evidence-based guidelines and primary research studies, rather than other review articles, unless they are systematic reviews.

The essential elements of this checklist are summarized in Table 3 .

Siwek J. Reading and evaluating clinical review articles. Am Fam Physician. 1997;55:2064-2069.

Shaughnessy AF, Slawson DC. Getting the most from review articles: a guide for readers and writers. Am Fam Physician. 1997;55:2155-60.

Pitt B, Zannad F, Remme WJ, Cody R, Castaigne A, Perez A, et al. The effect of spironolactone on morbidity and mortality in patients with severe heart failure. N Engl J Med. 1999;341:709-17.

Flynn CA, D'Amico F, Smith G. Should we patch corneal abrasions? A meta-analysis. J Fam Pract. 1998;47:264-70.

Slawson DC, Shaughnessy AF, Bennett JH. Becoming a medical information master: feeling good about not knowing everything. J Fam Pract. 1994;38:505-13.

Shaughnessy AF, Slawson DC, Bennett JH. Becoming an information master: a guidebook to the medical information jungle. J Fam Pract. 1994;39:489-99.

Slawson DC, Shaughnessy AF. Becoming an information master: using POEMs to change practice with confidence. Patient-oriented evidence that matters. J Fam Pract. 2000;49:63-7.

Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al. Methods Work Group, Third U.S. Preventive Services Task Force. Current methods of the U.S. Preventive Services Task Force. A review of the process. Am J Prev Med. 2001;20(3 suppl):21-35.

CATbank topics: levels of evidence and grades of recommendations. Retrieved November 2001, from: http://www.cebm.net/ .

Saha S, Hoerger TJ, Pignone MP, Teutsch SM, Helfand M, Mandelblatt JS. for the Cost Work Group of the Third U.S. Preventive Services Task Force. The art and science of incorporating cost effectiveness into evidence-based recommendations for clinical preventive services. Am J Prev Med. 2001;20(3 suppl):36-43.

Evidence-based medicine glossary. Retrieved November 2001, from: http://www.cebm.net/index.aspx?o=1116 .

Continue Reading

More in afp, more in pubmed.

Copyright © 2002 by the American Academy of Family Physicians.

This content is owned by the AAFP. A person viewing it online may make one printout of the material and may use that printout only for his or her personal, non-commercial reference. This material may not otherwise be downloaded, copied, printed, stored, transmitted or reproduced in any medium, whether now known or later invented, except as authorized in writing by the AAFP.  See permissions  for copyright questions and/or permission requests.

Copyright © 2024 American Academy of Family Physicians. All Rights Reserved.

Evidence-Based Practice PT

• Evidence-Based Practice
• Literature Review
• Developing a Topic
• Question Development
• Background questions
• Citation Management
• Critical Appraisal

What is a Literature Review

A literature review is a  systematic review of the published literature on a specific topic or research question designed to analyze-- not just summarize-- scholarly writings that are related directly to your research question .  That is, it represents the literature that provides background information on your topic and shows a correspondence between those writings and your research question.  This guide is designed to be a general resource for those completing a literature review in their field. 

Why a Literature Review is Important

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas.
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic.
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
• Indicates potential directions for future research.

A Literature Review Must:

A literature review must do these things

• be organized around and related directly to the thesis or research question you are developing
• synthesize results into a summary of what is and is not known
• identify areas of controversy in the literature
• formulate questions that need further research

A Literature Review is NOT

Keep in mind that a literature review defines and sets the stage for your later research.  While you may take the same steps in researching your literature review, your literature review is NOT:

• Not an annotated bibliography i n which you summarize each article that you have reviewed.  A lit review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.
• Not a research paper  where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

Types of Literature Reviews

D ifferent projects involve different  kinds  of literature reviews with different  kinds  and  amounts  of work. And, of course, the "end products" vary.

• Honors paper
• Capstone project
• Research Study
• Senior thesis
• Masters thesis
• Doctoral dissertation
• Research article
• Grant proposal
• Evidence based practice
• << Previous: Evidence-Based Practice
• Next: Developing a Topic >>
• The Interprofessional Health Sciences Library
• 123 Metro Boulevard
• Nutley, NJ 07110
• [email protected]
• Student Services
• Parents and Families
• Career Center
• Web Accessibility
• Visiting Campus
• Public Safety
• Disability Support Services
• Campus Security Report
• Report a Problem
• Login to LibApps

NUR 3710 Evidence Based Research Guide: Literature Review

• Evidence Based Practice
• Peer- Reviewed Journals

Literature Review

• About Zotero

A literature review is an essay or part of an essay that summarizes and analyzes research in a particular discipline. It assess the literature by reviewing a large body of studies on a given subject matter. It summarizes by pointing out the main findings, linking together the numerous studies and explaining how they fit into the overall academic discussion on that subject. It critically analyzes the literature by pointing out the areas of weakness, expansion, and contention. 

Literature Review Sections:

• Introduction: indicates the general state of the literature on a given subject.
• Methodology: states  where (databases), how (what subject terms used on searches), and what (parameters of studies that were included); so others may recreate the searches and explain the reasoning behind the selection of those studies.
• Findings: summary of the major findings in that subject.
• Discussion: a general progression from broader studies to more focused studies.
• Conclusion: for each major section that again notes the overall state of the research, albeit with a focus on the major synthesized conclusions, problems in the research, and even possible avenues for further research.
• References: a list of all the studies using proper citation style. 

Literature Review Tips:

• Beware of stating your own opinions or personal recommendations (unless you have evidence to support such claims).
• Provide proper references to research studies.
• Focus on research studies to provide evidence and the primary purpose of the literature review.
• Connect research studies with the overall conversation on the subject.
• Have a search strategy planner and log to keep you focused.

Literature reviews are not book reports or commentaries; make sure to stay focused, organized, and free of personal biases or unsubstantiated recommendations.

Literature Review Examples:

• Lemetti, T., Stolt, M., Rickard, N., & Suhonen, R. (2015). Collaboration between hospital and primary care nurses: a literature review. International Nursing Review , 62 (2), 248-266. doi:10.1111/inr.12147  

Templates for Starting Your Literature Review

• Literature search strategy planner
• Literature search log template
• Review Literature

Featured Ebook

Literature Review Steps

1. Choose a topic and define your research question.

Your literature review should be guided by a focus research question.   Consider PICO and FINER criteria for developing a research question.

• Make sure your research question is not too broad or too narrow. Do a couple of pre-searches to see what information is out there and determine if it is a manageable topic.
• Identify the main concepts of your research question and write down terms that are related to them. Keep a list of  terms that you can use when searching.
• If possible, discuss your topic with your professor. 

2. Decide on the scope of your review.

Check with your assignment requirements and your professor for parameters of the Literature Review.

• How many studies are you considering?
• How comprehensive will your literature review be?
• How many years should it cover? 

3. Select appropriate databases to search.

Make a list of the databases you will search. 

• Don't forget to look at books, dissertations or other specialized databases .
• Contact your librarian to make sure you are not missing any vital databases for that topic.

4. Conduct searches and find relevant literature.

As you are searching in databases is important to keep track and notes as you uncover information.

• Read the abstracts of research studies carefully instead of just downloading articles that have good titles.
• Write down the searches you conduct in each database so that you may duplicate or avoid unsuccessful searches again.
• Look at the bibliographies and references of research studies you find to locate others .
• Look for subject terms or MeSH terms that are associated with the  research studies you find and use those terms in more searches.
• Use a citation manager such as Zotero or Endnote Basic to keep track of your research citations.

5. Review the literature. 

As you are reading the full articles ask the following questions when assessing studies:

• What is the research question of the study?
• Who are the author(s)? What are their credentials and how are they viewed in their field?
• Has this study been cited?; if so, how has it been analyzed?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.  Does the research seem to be complete?  What further questions does it raise?
• Are there any conflicting studies; if so why?

Throughout the process keep careful notes of your searches and findings so it is easier to put it together when it comes to the writing part.

• << Previous: Databases
• Next: Writing >>
• Last Updated: Jul 15, 2024 2:54 PM
• URL: https://hpu.libguides.com/NUR3710

• UWF Libraries

Evidence Based Nursing

• Types of Reviews
• What is Evidence -Based Practice & PICO
• PICO Widgets for Searching
• PubMed - Info & Tutorials
• Cochrane Library - Info & Tutorials
• Joanna Briggs Institute - Tutorial
• Finding the Evidence: Summaries & Clinical Guidlines
• Find Books & Background Resources
• Statistics and Data
• APA Formatting Guide
• Citation Managers
• Library Presentations

What are the types of reviews?

As you begin searching through the literature for evidence, you will come across different types of publications. Below are examples of the most common types and explanations of what they are. Although systematic reviews and meta-analysis are considered the highest quality of evidence, not every topic will have an Systematic Review or Metanalysis.

Use the PRISMA Online Checklist to assess research and systematic reviews

Literature Review Examples

Remember, a literature review provides an overview of a topic. There may or may not be a method for how studies are collected or interpreted. Lit reviews aren't always obviously labeled "literature review"; they may be embedded within sections such as the introduction or background. You can figure this out by reading the article . 

• Dance therapy for individuals with Parkinson's Disease Notice how the introduction and subheadings provide background on the topic and describe way it's important. Some studies are grouped together that convey a similar idea. Limitations of some studies are addressed as a way of showing the significance of the research topic.
• Ethical Issues Regarding Human Cloning: A Nursing Perspective Notice how this article is broken into several sections: background on human cloning, harms of cloning, and nursing issues in cloning. These are the themes of the different articles that were used in writing this literature review. Look at how the articles work together to form a cohesive piece of literature.

Systematic Review Examples

Systematic reviews address a clinical question.  Reviews are gathered using a specific, defined set of criteria.

• Selection criteria is defined
• The words "Systematic Review" may appear int he title or abstract
• BTW -> Cochrane Reviews aka Systematic Reviews
• Additional reviews can be found by using a systematic review limit 
• A Systematic Review of Animal-Assisted Therapy on Psychosocial Outcomes in People with Intellectual Disability
• The determinants and consequences of adult nursing staff turnover: a systematic review of systematic reviews
• Cochrane Library (Wiley) This link opens in a new window Over 5000 reviews of research on medical treatments, practices, and diagnostic tests are provided in this database. Cochrane Reviews is the premier resource for Evidence Based Practice.
• PubMed (NLM) This link opens in a new window PubMed comprises more than 22 million citations for biomedical literature from MEDLINE, life science journals, and online books.

Meta-Analysis Examples

Meta-analysis is a study that combines data from OTHER studies. All the studies are combined to argue whether a clinical intervention is statistically significant by combining the results from the other studies.  For example, you want to examine a specific headache intervention without running a clinical trial.  You can look at other articles that discuss your clinical intervention, combine all the participants from those articles, and run a statistical analysis to test if your results are significant. Guess what? There's a lot of math. 

• Include the words "meta-analysis" or "meta analysis" in your keywords
• Meta-analyses will always be accompanied by a systematic review, but a systematic review may not have a meta-analysis
• See if the abstract or results section mention a meta-analysis
• Use databases like Cochrane or PubMed
• Exercise Interventions for Preventing Falls Among Older People in Care Facilities: A Meta-Analysis
• Acupuncture for the prevention of tension-type headache This is a systematic review that includes a meta-analysis. Check out the Abstract and Results for an example of what a meta-analysis looks like!
• << Previous: What is Evidence -Based Practice & PICO
• Next: Finding the Evidence >>
• Last Updated: May 28, 2024 1:18 PM
• URL: https://libguides.uwf.edu/EBN

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Systematic reviews: the heart of evidence-based practice

Affiliation.

• 1 Academic Center for Evidence-Based Nursing, University of Texas, Health Science Center at San Antonio, 7703 Floyd Curl Drive MC 7951, San Antonio, TX 78229-3900, USA. [email protected]
• PMID: 11759425
• DOI: 10.1097/00044067-200111000-00009

Research utilization approaches in nursing recently have been replaced by evidence-based practice (EBP) approaches. The heart of the new EBP paradigm is the systematic review. Systematic reviews are carefully synthesized research evidence designed to answer focused clinical questions. Systematic reviews (also known as evidence summaries and integrative reviews) implement recently developed scientific methods to summarize results from multiple research studies. Specific strategies are required for success in locating systematic reviews. Major sources of systematic reviews for use by advanced practice nurses in acute and critical care are the Online Journal of Knowledge Synthesis for Nursing, Agency for Healthcare Research and Quality, and the Cochrane Library. This discussion describes systematic reviews as the pivotal point in today's paradigm of EBP and guides the advanced practice nurse in locating and accessing systematic reviews for use in practice.

PubMed Disclaimer

Publication types

• Search in MeSH

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.
• Wolters Kluwer

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Social Work 795: Evaluation of Social Work Practice and Programs

• What is Evidence-Based Practice?
• Create a search strategy
• Find Evidence-Based Interventions
• Research methods
• Cite sources in APA
• Citation Managers

Evidence-Based Practice (EBP)

NASW states that social work EBP is "a process involving creating an answerable question based on a client or organizational need, locating the best available evidence to answer the question, evaluating the quality of the evidence as well as its applicability, applying the evidence, and evaluating the effectiveness and efficiency of the solution."

Read more from NASW about EBP in Social Work

EBP Tutorial Duke University Medical Center Library

Learn more about evidence synthesis reviews

Levels of Evidence

• The Pyramid
• Non-Research Information

Observational Studies

Experimental studies.

• Critical Analysis

Levels of evidence

The levels of evidence pyramid demonstrates a hierarchy of information sources based on the strength of the evidence reported. Click through the tabs to learn more about the each of the levels and the strength of the evidence and example research articles for different study types.

Non-Evidence-based sources

While these information sources do not meet the criteria for evidence, this kind of information can help you to get background information or context on a particular topic area, are typically easier to understand, and may include references to evidence-based research.

• Non-Evidence-based Expert Opinions : These can include commentary statements, speeches, or editorials written by prominent experts asserting ideas that are reached by conjecture, casual observation, emotion, religious belief, or ego
• Non-EBP guidelines : Practice guidelines that exist because of eminence, authority, eloquence, providence, or diffidence based approaches to healthcare
• News Articles : News articles are written by journalists for the general public and may report on or summarize research studies and outcomes
• Editorials : Opinions written by experts, non-experts, or regular folks that are published by news outlets, magazines, or academic journals
• Commentary : similar to an editorial, but it may be identified as a commentary, which can be an invited informal and non-reviewed short article pertaining to a particular concept or idea.
• Narrative literature review articles : Non-systematic and non-exhaustive survey of the literature on a specific topic. However, evidence synthesis review articles are considered to be a high degree of evidence by nature of their methodology (see critical appraisal tab).

Let's Talk about Review Articles

Review articles are common in health literature. They are typically overviews of literature found on topics, but do not go so far as to meet the methodological requirements for a Systematic Review.

These articles may contain some critical analysis, but will not have the rigorous criteria that a Systematic Review does. They can be used to demonstrate evidence, albeit they do not make a very strong case as they are secondary articles and not originally conducted observational or experimental research.

These types of publications have the lowest evidence strength in the hierarchy. The evidence is largely anecdotal since they often lack a systematic methodology, have limited statistical sampling, even if the studies are in some instances empirical and verifiable. Examples of observational studies are:

• example: Thoele, K., Ferren, M., Moffat, L. et al. (2020). Development and use of a toolkit to facilitate implementation of an evidence-based intervention: a descriptive case study. Implement Sci Commun, 1(86). DOI: 10.1186/s43058-020-00081-x
• example: Tone, J., Chelius, B. & Miller, Y.D. (2022). The effectiveness of a feminist-informed, individualised counselling intervention for the treatment of eating disorders: a case series study. J Eat Disord , 10(70). DOI: 10.1186/s40337-022-00592-z
• Choi, S., Bunting, A., Nadel, T., Neighbors, C. J., & Oser, C. B. (2023). Organizational access points and substance use disorder treatment utilization among Black women: a longitudinal cohort study. Health & Justice , 11 (1), 1–12. DOI: 10.1186/s40352-023-00236-7
• Example: Belenko, S., Dennis, M., Hiller, M., Mackin, J., Cain, C., Weiland, D., Estrada, B., & Kagan, R. (2022). The impact of juvenile drug treatment courts on substance use, mental health, and recidivism: Results from a multisite experimental evaluation. The Journal of Behavioral Health Services & Research, 49 (4), 436-455. DOI: 10.1007/s11414-022-09805-4
• Example:  Putnam-Hornstein, E., Prindle, J., & Hammond, I. (2021). Engaging Families in Voluntary Prevention Services to Reduce Future Child Abuse and Neglect: a Randomized Controlled Trial. Prevention Science , 22 (7), 856–865. DOI: 10.1007/s11121-021-01285-w

Critical Appraisal

"Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context." Burls, A. (2009). What is critical appraisal? In What Is This Series: Evidence-based medicine. Available online at What is Critical Appraisal?

Examples of Critical Appraisal

Evidence synthesis reviews are types of critical appraisal. Examples of evidence synthesis reviews are scoping reviews, systematic reviews, and meta-analysis. To find these types of articles, search for "systematic review", "scoping review", or meta-analysis in the title. Learn more about conducting evidence synthesis reviews .

• Example:  Kokorelias, K. M., PhD., Shiers-Hanley, J., Li, Z., & Hitzig, S. L., PhD. (2023/10//). A Systematic Review on Navigation Programs for Persons Living With Dementia and Their Caregivers. The Gerontologist, 63 (8), 1341. DOI: 10.1093/geront/gnac054

​ Example:  Hans, B. B., Drozd, F., Olafsen, K., Nilsen, K. H., Linnerud, S., Kjøbli, J., & Jacobsen, H. (2023/08//). The effect of relationship-based interventions for maltreated children and adolescents: A systematic review and meta-analysis. Development and Psychopathology, 35 (3), 1251-1271. doi: 10.1017/S0954579421001164

• Example:  After thorough testing and experimentation, researchers, doctors, and product developers created and started using less-invasive oxygen monitoring devices to improve recovery times after surgeries. These are now standard equipment.

Tips for identifying empirically based research

Characteristics to look for:

States the problem, population, or research question under study

Defines the group or issue being studied

Study methodology is reported

Alternative interventions may be included or compared

May be quantitative or qualitative   [check with your course instructor or syllabus, as the course focus may be on just one or the other]

May include tests or surveys (embedded, as an appendix, or referred to by proper name)

May be reproducible; to be replicated or adapted to a new study

Tips for searching for empirically based research

Search for peer-reviewed journal articles that report research findings in one of the recommended databases. Some databases have a filter or advanced search limiter focus results on empirical research, for example filters for systematic reviews or randomized control trials . If a filter/limiter is not available, enter keywords to match on appropriate content and/or to look for these terms in the abstract or article itself:

• methods or methodology
• quantitative
• statistic* ( the asterisk is used as a "wildcard" ending to your search term which allows the database to match on statistic, statistics or statistical)
• [inclusion of] charts, statistical tables or graphs
• << Previous: Welcome
• Next: Create a search strategy >>
• Last Updated: Jul 16, 2024 3:13 PM
• URL: https://guides.library.uwm.edu/SW795

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Springer Nature - PMC COVID-19 Collection

Literature-based discovery approaches for evidence-based healthcare: a systematic review

Sudha cheerkoot-jalim.

1 Department of Information and Communication Technologies, University of Mauritius, Reduit, Mauritius

Kavi Kumar Khedo

2 Department of Digital Technologies, University of Mauritius, Reduit, Mauritius

Associated Data

Not applicable.

Not applicable

Literature-Based Discovery (LBD) is a text mining technique used to generate novel hypotheses from vast amounts of literature sources, by identifying links between concepts from disparate sources. One of the main areas where it has been predominantly applied is the healthcare domain, whereby promising results, in the form of novel hypotheses, have been reported. The purpose of this work was to conduct a systematic literature review of recent publications on LBD in the healthcare domain in order to assess the trends in the approaches used and to identify issues and challenges for such systems.

The review was conducted following the principles of the Kitchenham method. The selected studies have been scrutinized and the derived findings have been reported following the PRISMA guidelines.

The review results reveal useful information regarding the application areas, the data sources considered, the approaches used, the performance in terms of accuracy and reliability and future research challenges. The results of this review will be beneficial to LBD researchers and other stakeholders in the healthcare domain, by providing them with useful insights on the approaches to adopt, data sources to consider, evaluation model to use and challenges to reflect on.

The synthesis of the results of this work has shed light on recent issues and challenges that drive new LBD models and provides avenues for their application in other diverse areas in the healthcare domain. To the best of our knowledge, no such recent review has been conducted.

Introduction

Healthcare management, being one of the highest priorities of most governments, attracts huge investments in terms of health and medical research worldwide. Medical research was found to be the main contributing factor in the improvement of health and longevity of individuals and populations in developed countries [ 1 ]. Researchers in the field are making new discoveries and generating knowledge, which has the potential to enhance healthcare delivery, improve patient health outcomes and reduce healthcare costs, thus strengthening the overall healthcare system and economy. This is only achievable if the knowledge is actually put into action [ 2 ]. However, the transfer of research findings into healthcare practice in the clinical setting, known as knowledge translation [ 3 ], is a very complex and slow process, often resulting in patients not being provided with the most appropriate care, although better treatment recommendations have been proposed and demonstrated. A frequently stated average time lag for knowledge translation is 17 years [ 4 ]. Understanding the various stages of knowledge translation and speeding up the process is a policy priority for many health research systems [ 4 ].

In order to leverage new medical research findings more quickly for the benefit of patients, medical practitioners are encouraged to adopt the practice of evidence-based medicine, whereby medical practitioners are expected to scrutinize the scientific and clinical research literature in their respective areas in an attempt to translate health research knowledge into effective healthcare action more quickly. However, due to the large volumes of biomedical literature available and the time constraints of medical practitioners, the practice of evidence-based medicine has become a major challenge [ 5 ]. This limitation can be considerably overcome by the use of appropriate computation techniques for the automated or semi-automated knowledge extraction from relevant research literature. A broad term commonly used for such techniques is literature based discovery (LBD), whose main goal is to generate novel hypotheses from the vast available biomedical literature by discovering unknown associations in existing knowledge [ 6 ]. Recent advances in machine learning, text mining and statistical analysis techniques have spurred research in this field and have resulted in many publications on the design and application of LBD systems for various use cases in the biomedical and healthcare domains.

The purpose of this work is to perform a systematic literature review of recently published research papers on the application of LBD for evidence-based healthcare, with the objective of identifying and integrating the findings of the most relevant individual studies. It is expected that the results of this review will give insights on the different LBD approaches and tools used in various application areas in the healthcare domain. It will help establish to what extent research has progressed in the field, with a focus on performance criteria like effectiveness, accuracy and reliability. A main outcome would be to identify research challenges, which will invoke further studies and thus, provide avenues for future research in other areas in the healthcare domain. The Kitchenham guidelines for performing systematic literature reviews [ 7 ] was adopted and the reporting of this paper follows PRISMA (preferred reporting items for systematic reviews and meta-analysis) guidelines [ 8 ]. To the best of our knowledge, no such recent review has been performed for evidence-based healthcare.

Evidence-based healthcare

The challenges of knowledge translation have become a major concern to individuals who seek and need healthcare, healthcare providers, policy makers and funders of health services. The incorporation of scientific medical discoveries into practice guidelines and policies in the clinical setting can greatly improve healthcare delivery and patient health outcomes, and is the basis of evidence-based healthcare [ 9 ]. Evidence-based practice involves clinical decision making which considers the best and most up-to-date available scientific evidence, together with patient values and preferences, the clinical judgment of the medical practitioner and the context in which the care is provided [ 10 ]. Healthcare professionals seek evidence to support and justify any activity or intervention for patient care.

Literature based discovery in healthcare

In their practice of evidence-based medicine, medical practitioners are expected to scrutinize the best available evidence for making decisions about the care of individual patients. However, with the increasing volume of academic research papers and related structured knowledge resulting from medical research worldwide, they only focus on publications that are directly relevant to their respective area of specialization and often skip other potentially relevant research. Thus, discoveries in one field remain unknown to others and potential connections between sub-fields are often missed out [ 11 ]. This limitation can be greatly curbed by LBD, which can automate or semi-automate the analysis of online resources from disparate sources to find new discoveries. With the exponential growth of scientific literature, LBD is becoming an increasingly important tool for facilitating research [ 12 ].

LBD generates discoveries not yet published anywhere, by combining knowledge extracted from varied literature sources and therefore, supports hypothesis generation [ 13 ]. There are two modes of discovery in LBD, namely open discovery and closed discovery. Open discovery starts with a concept X and tries to generate a potential association between X and another concept Z, based on an intermediate concept Y. This follows from the ABC co-occurrence model, which states that if A and B are often associated to each other, and B and C are also often associated to each other, there may potentially be an association between A and C, even if this association is not mentioned in any research paper [ 14 ]. In contrast, in closed discovery, both the start concept X and end concept Z are known, and an association between X and Z is predicted, based on a hypothesis about the relationship between X and Z. This technique then attempts to demonstrate the hypothesis through an intermediate concept Y.

LBD approaches in healthcare are becoming essential, since biomedical knowledge is spread out across a larger number of publications [ 15 ]. Potential discoveries in healthcare can be associations that exist between biomedical concepts, which are not usually discussed together in the literature. Appropriate implementation of LBD techniques have the potential to predict future strong associations between these concepts [ 15 ] and therefore entails further research. In the LBD approach the starting concept X may be a disease and the end concept Z may be a treatment or cause for the disease. The results of such discoveries need to be further investigated through experimental methods or clinical studies.

Materials and methods

This review has been performed following the guidelines on undertaking systematic literature reviews by Kitchenham and Charters [ 7 ] and the reporting follows the PRISMA guidelines [ 8 ]. The methodology consisted of first setting out the research questions to give a focus for this review, followed by the specification of the search strategy, the application of assessment criteria for the selection of papers and finally the data analysis and extraction.

Research questions

Based on the objectives of this review, the research questions have been set out and elaborated as follows:

RQ1: What are the main application areas of literature based discovery in evidence-based healthcare?

We seek to find out the different application areas in which the application of LBD techniques has proved to be successful in the healthcare domain.

RQ2: Which important/impactful literature sources are considered by researchers/practitioners for literature based discovery?

The foundation of LBD is the large amount of scientific literature available for a specific field of study. It is therefore important to identify the different literature sources which have been harnessed for LBD in the different studies.

RQ3: Which specific literature based discovery approaches and tools have proven to be effective in the healthcare domain?

Due to the peculiarity of the healthcare domain, LBD techniques have to be adapted to specific application areas. There is therefore the need to investigate the specific LBD techniques/approaches which are more relevant and effective for the healthcare domain.

RQ4: How do literature based discovery systems in the healthcare domain perform in terms of accuracy and reliability?

Accuracy and reliability are imperative evaluation criteria for any computational technique in the healthcare domain, since a wrong intervention can lead to harmful consequences for the patient. We therefore study the different evaluation strategies used for LBD systems and find out their performance in terms of accuracy and reliability.

Search strategy and study selection

The search strategy involved the identification of potential research papers to be included in the review by performing a search on Google Scholar, with keywords ‘“Literature-based discovery” in health’. Google Scholar was chosen since it indexes scientific articles from various scholarly publishers and professional societies like Springer, ScienceDirect, ACM, IEEE Xplore, ResearchGate amongst others [ 16 ]. It also indexes biomedical-specific journals like the Journal of Biomedical Informatics, PLOS ONE and BioMed Central (BMC). Gusenbauer [ 17 ] performed a comparative study of academic search engines in 2019 and concluded that “Google Scholar is currently the most comprehensive academic search engine”. Keyword search was then followed by a manual screening of reference lists of relevant primary studies to extend the search space.

Eligibility criteria

Based on the objectives of this systematic review, we have set some inclusion and exclusion criteria to guide the study selection process, as follows. The focus of this review being on recent advances in LBD techniques and approaches, we considered studies carried out during the last five years, that is, since 2015. We only considered peer-reviewed papers published in the English language. Primary studies were included while secondary and tertiary studies, like surveys, systematic reviews and meta analyses were excluded. During an initial screening of studies, we came across papers which describe general LBD techniques without showing their application in the healthcare domain. Such studies were not included, since the objective of this review was to get insights on the different approaches which are more appropriate for specific application areas of LBD. We thus considered papers which describe the use of LBD approaches in a specific application area in the healthcare domain.

The database search was performed on 2 nd February 2021. The keyword search returned 650 results, after applying the filter on year of publication. The manual screening of reference lists of relevant studies returned 12 eligible studies. 8 duplicate studies were identified from the two sources, resulting in 654 studies to screen. After a rigorous screening of the titles and abstracts based on the inclusion and exclusion criteria, 29 studies were pre-selected for the review.

Quality assessment

After initial screening based on the inclusion and exclusion criteria, the pre-selected studies were assessed for “quality” in order to integrate more detailed inclusion and exclusion criteria. Based on the research questions, four quality assessment criteria were set as shown in Table ​ Table1. 1 . The possible outcomes for each criteria were “Yes” if the paper met the criteria and “No” if it did not meet the criteria. Two of the quality assessment criteria also had a “Partially” outcome.

Quality Assessment Criteria

During the quality assessment phase, appropriate scores were given to each pre-selected study. A score of 1 was given for a “Yes” outcome, 0 for a “No” outcome and 0.5 for a “Partially” outcome. Studies which obtained a score of at least 2.5 were included in the final review. This would allow for one “No” and one “Partially” outcome in the outmost scenario. After the quality assessment phase, 23 studies have been selected for the final review, based on the scores obtained. Figure  1 shows the PRISMA flow diagram for the study selection process.

PRISMA Flow Diagram for the Study Selection Process

The selected studies were thoroughly analyzed with an objective to extract information which would give insights to the research questions. More particularly, the information extracted were: the medical application area in which LBD was utilized and the discovery made as a result of LBD, the literature source/s considered, the type of discovery (open or closed), the techniques and tools used in the LBD approach, the performance of the system and the challenges identified by the authors. The data synthesis is shown in Table ​ Table2 2 .

Data Synthesis

N/A means Not Available

The selected studies were scrutinized with a major focus on the objectives of this review. The work of the various authors and their findings were mapped to the research questions and are discussed in the following sub-sections.

From the studies analyzed, it was found that LBD techniques have been implemented in a myriad of application areas in the healthcare domain, as described below.

Drug repurposing

Drug repurposing is one main application area in which researchers have put efforts, mostly because of the promising results achieved by the different LBD approaches proposed. Due to the huge costs and excessive amount of time involved in developing new drugs, it is regarded as a better alternative. Several studies [ 18 , 19 , 21 , 23 , 25 ] generated a list of potential drug-disease pairs by using drug-gene and gene-disease semantic predications. Phenotypes and symptoms have also been used as the linking concept between drug and disease [ 16 ]. Some studies have used knowledge-graph based drug discovery methods [ 18 – 20 ].

Pharmacovigilance and drug interactions

Pharmacovigilance involves the continuous monitoring of drug safety after drugs are put on the market, which is necessary since some adverse drug events (ADEs) remain undetected during clinical trials and unreported in adverse event reporting systems such as FAERS (FDA Adverse Event Reporting System). The health hazards that ADEs may pose to individuals motivate the extensive work on the application of various computational methods for pharmacovigilance. Authors of this study have either used an open LBD [ 15 , 23 , 24 ] or a closed LBD [ 22 , 25 ] approach for the detection of drug/ADE pairs.

Identification of potential causes, therapies or treatments for specific diseases

LBD’s potential to contribute to the advancement of the medical field has been demonstrated by the development of text mining systems which have been able to identify possible causes, therapies or treatments for specific diseases. Discoveries about connections between diet and degenerative diseases [ 34 , 35 ] were made from scientific literature to support better understanding and treatment of such diseases. LBD techniques have been used for rehabilitation therapy repositioning for stroke [ 31 ] and treatment repurposing for inflammatory bowel disease [ 36 ]. Other discoveries were made in the area of cancer [ 32 ] and chronic kidney disease [ 33 ].

Explanation for the correlation between diseases

Disease comorbidity is very common and is a popular area of research in the medical community, because of its impact on the treatment of diseases. Knowledge of the association between diseases can significantly improve the understanding of the mechanisms of diseases, thus aiding in better prevention and treatment [ 37 ]. Thus, Chen et al. [ 37 ] have used an open LBD approach for the detection of associations among complex diseases. Closed LBD approach was also used for the explanation of the correlation between epilepsy and inflammatory bowel disease [ 38 ], and between myocardial infarction and depression [ 39 ]. Rather et al. [ 40 ] proposed the use of deep learning for the discovery of potential new biomedical knowledge .

Table ​ Table3 3 summarizes the main application areas and the number of studies for each.

Main application areas for LBD in healthcare

RQ2: Which important/impactful literature sources are considered by researchers / practitioners for literature based discovery?

The main literature sources for LBD leveraged by authors of studies in this review are Medline (10 studies) and PubMed (13 studies). Medline, the bibliographic database of the US National Library of Medicine, is indexed with Medical Subject Headings (MeSH) terms, making search in the biomedical domain more effective. This explains its popularity among LBD researchers. PubMed on the other hand, is an interface to search Medline together with other additional biomedical content. Tools which extract data from PubMed and Medline, like Global Network of Biomedical relationships (GNBR) [ 20 ] and Semantic Medline Database (SemMedDB) [ 21 , 26 ] have also been proposed. Apart from PubMed, Zhang et al. [ 25 ] also extracted data from CORD-19 (Covid-19 Open Research Dataset), which contains Covid-19-related literature, which may not yet be available on PubMed. An additional literature source, Chinese Science Database (CNKI), was used to extract herb-disease pairs in Traditional Chinese Medicine [ 26 ].

Since the data sources mainly consist of free-text, the main techniques behind LBD are text mining and natural language processing (NLP). Most LBD approaches proposed have extracted meanings from biomedical text by using Unified Medical Language System (UMLS) concepts and MeSH terms. The approaches used by authors of studies in this review are broadly categorized and described below.

Co-occurrence-based models

The ABC model of LBD is a common relation extraction technique used by many authors [ 18 – 20 , 26 , 30 , 31 , 39 ]. The associations between the different concepts are usually deduced from semantic predications extracted from NLP tools, like SemRep and MetaMap, which have been the most preferred tools. If the output of the ABC method consists of a long list of C terms, then these are ranked based on specific criteria and the higher-ranked C terms are considered as plausible hypotheses. Co-occurrence-based metrics are often used for analyzing the strength of entity associations, and prioritization of C terms are often based on the total frequency of co-occurrence [ 32 ]. Furthermore, Gubiani et al. [ 35 ] proposed a method to identify outlier documents by making use of two tools, namely OntoGen for outlier document detection and CrossBee for cross domain exploration.

Table ​ Table4 4 shows the different biomedical concepts A, B and C which have been considered in the studies in this review.

Biomedical concepts A, B and C considered in the ABC model of LBD

Distributional models

While most LBD methods apply co-occurrence-based methods to assess the relatedness of biomedical concepts, distributional models are also widely used. These models build vector representations of concepts which are based on the context in which they appear in literature. Relatedness between a pair of concepts is then derived based on the similarity between the vectors. Various distributional semantic techniques which have been proposed include Semantic Predications [ 18 , 25 , 30 ], Latent Semantic Analysis (LSA) [ 37 ], Predication-based Semantic Indexing (PSI) [ 28 ] and composite feature vectors [ 29 ]. Mower et al. [ 29 ] have shown that distributional models perform better than co-occurrence-based models.

Machine Learning models

Several authors have used machine learning in different steps of their LBD methodology. For text analysis, Pyysalo et al. [ 32 ] propose the use of machine learning-based methods for the recognition of biomedical entity names and their grounding to domain-specific ontology identifiers. Ranking of LBD-generated hypotheses have been performed by Zhang et al. [ 27 ] through a machine learning-based filter (lasso regression filter) and Rastegar-Mojarad et al. [ 21 ] by using a binary classifier. Machine learning algorithms like logistic regression [ 22 , 23 , 29 ] and k-Nearest Neighbor (kNN) [ 29 ] have been incorporated in models proposed by authors in this review. Rather et al. [ 40 ] integrated Word2vec, a neural network based algorithm, in their LBD approach and showed that the model was able to retrieve strong relationships which were not identified by UMLS. Deep learning has also been used in LBD techniques[ 18 , 35 ].

Knowledge-graph models

Knowledge-graph models use graph theory to identify novel associations among various concepts. In their LBD approach for drug discovery, Zhao et al. [ 22 ] constructed a biomedical knowledge graph based on semantic predications. A path ranking algorithm was then used to extract drug-disease relation path features. Sang et al. [ 23 ] also use a knowledge graph-based drug discovery method, which involves the training of a logistic regression model by learning the semantic types of paths in the knowledge graph. Knowledge graph embedding and knowledge graph completion have also been used [ 24 , 25 ].

The papers analyzed have shown that diverse performance evaluation methods have been used for LBD systems, mostly due to the peculiarities of the healthcare domain and the specific requirements of the varied application areas.

No gold standard to benchmark performance

The evaluation of LBD systems in terms of accuracy and reliability is quite challenging in the healthcare domain. It becomes difficult for researchers to reliably distinguish between false positive signals and new discoveries. Most authors therefore have to rely on manual review by experts to confirm the final candidates for LBD. Many authors have claimed that there was no gold standard against which they could accurately benchmark the performance of their approaches [ 18 , 21 ] and that precision and recall were not good metrics to measure the performance in all conditions [ 20 ].

Accuracy and reliability impacted by performance of text mining tools

The performance of the systems developed in several studies of this review is highly impacted by the performance of the tools and resources used in the LBD approach. In the evaluation of their system, Rastegar-Mijarad et al. [ 18 ] used the Comparative Toxicogenomics Database (CTD) resource, which does not annotate the type of relationship between drug and disease, therefore resulting in loss of valuable information. Sources of error are also often introduced in text mining tools like SemRep, due to inaccuracies in language processing or in the literature itself [ 21 , 22 , 25 , 27 , 38 ] and MetaMap whose accuracy reduces in the presence of ambiguity, resulting in the inability to resolve word sense disambiguation [ 23 ]. Sosa et al. [ 24 ] have acknowledged that the performance of their algorithm could considerably be improved if NLP tools improved their capability to capture complex relationships from unstructured text.

Computationally intensive models

The resource requirements for most LBD systems, specially those which use the open discovery approach are huge. Therefore, it is quite challenging for researchers to make their model computationally feasible, thereby imposing certain limitations resulting in suboptimal outcomes [ 24 , 25 , 28 , 32 ]. One limitation of Pyysalo et al.’s [ 32 ] open discovery method is that it can recognize only a single correct target response for each case and their system is currently limited to discovery over paths of length two. Since the graph generated by the relations in SemMedDB is very large, making models computationally intensive, Zhang et al. [ 25 ] have used a sub-graph instead which resulted in loss of information, therefore affecting the accuracy of their model.

Limited data sets

Many authors agree that the use of larger and more variate data sets would improve the accuracy of their models. Limitations encountered include the use of unbalanced [ 21 ] and small [ 22 , 32 ] data sets. The models proposed by Zhao et al.[ 22 ] and Pyysalo et al. [ 32 ] perform well using a rather small data set. However, the authors agree that their system’s computational efficiency may be greatly reduced if the knowledge base is large. Yang et al. [ 19 ] believe that the rankings of the drug-disease pairs generated by their model may be adversely affected since their methodology did not consider aliases for drug names.

Research challenges and future directions

The proposed LBD approaches have demonstrated considerable achievements and promising results in the discovery process. An in-depth analysis of the techniques used has revealed major insights to the main research challenges and future directions for such systems. The proper handling of the research challenges will definitely result in improved accuracy and performance in the LBD process.

Minimize manual expert review

From the analysis of the various studies, it was found that extensive manual expert review was required for the selection of the final LBD candidates from a very large number. There is therefore the need to develop approaches to prioritize LBD candidates, which will provide domain experts with essential evidence instead of information overload. The following approaches are proposed to decrease the effort required by domain experts:

• Determine a suitable threshold score for LBD candidates [ 18 , 21 ]. Candidates below that threshold would be considered as false positives and those above the threshold would be considered for further investigations and experiments.
• Develop a tool to provide recommendations for hypothesis generation [ 35 ]
• Make use of rigorous statistical techniques to replace the manual review step by a more automated approach [ 18 ]
• Design NLP techniques to detect false predications which occur due to negative associations [ 19 – 21 , 23 ]

Seamless integration of multiple data sources for improved accuracy

Most models designed have only considered PubMED and MEDLINE abstracts as their main text corpus. Many authors have proposed the incorporation of additional data sources as the text corpus of their models to improve accuracy. A larger knowledge base has the potential to produce more complex relation paths. The additional data sources which could be considered include:

• NIH grants summary to identify potentially hidden and novel associations by investigating exploratory analysis methods [ 40 ]
• Biological data to find more drug candidates for Covid-19 drug repurposing [ 25 ]
• Biomedical ontologies to consider additional interesting associations [ 38 ]
• Drug-disease databases like CTD and DrugBank for better training in drug-repurposing [ 19 ]
• FAERS data for pharmacovigilance methods instead of only relying on EHR data [ 28 ]
• Spontaneous reporting data for the extraction of drug-side effect associations [ 29 ]

Computational optimisation for improved accuracy and reliability

Studies in this review have clearly indicated the quest for researchers to obtain more accurate results. Due to the very large datasets and the multitude of possible pathways, the LBD models proposed are computationally intensive, therefore leading to certain limitations. Techniques proposed to improve accuracy include:

• Integration of machine learning and deep learning algorithms in LBD models [ 27 , 29 , 37 ] ,
• Development of high-quality NLP tools for better accuracy, due to the reported shortcomings of existing tools
• Use of relevant tools for the normalization of gene and disease targets [ 19 ]
• Consideration of full texts of research articles instead of only titles and abstracts [ 32 ]
• Use of graph embedding to obtain long paths [ 23 ]
• Consideration of indirect relationships from knowledge graphs [ 24 ]

The purpose of this work was to carry out a systematic literature review of recent publications in Literature Based Discovery approaches in the field of evidence-based healthcare. Four research questions had been set out in the planning phase of the review and the papers were deeply analyzed so as to get insights on the research questions. This work has revealed the potential of LBD techniques to discover hidden knowledge in emerging areas of healthcare and provides a comprehensive contextualization to various stakeholders in the health informatics community. The results of this review will therefore help the latter to have a good understanding of the appropriate approaches used in different application areas and contexts, and the challenges they will have to face.

The synthesis of the results of this work has shed light on recent issues and challenges that drive new LBD models and provides avenues for their application in other diverse areas in the healthcare domain. The research challenges identified show different perspectives to address further research in the field and, if properly tackled, will result in better overall accuracy and performance of LBD systems, therefore contributing in the speeding up of the knowledge translation process.

Authors’ Contributions

All authors have made a substantial, direct, intellectual contribution to this study.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability

Code availability, declarations.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

• Site Search
• MU Libraries Gateway
• Ellis Library
• Engineering Library
• Veterinary Medical Library
• More Libraries & Collections
• Distance Education Students
• Medical Students
• Nursing Students and Nurses
• Patients & Everyday Health
• Borrow & Checkout
• Electronic and Print Reserves
• Expert Search Service
• Interlibrary Loan and Document Delivery
• Photocopying
• Request a Library Purchase
• Room Reservations
• Citation Styles and Tools
• Evidence Based Medicine
• Find Articles
• Mobile Medical Resources
• Research by Subject
• Workshops & Videos
• Giving Opportunities
• Maps/Directions
• Staff Directory
• University of Missouri Libraries
• Library Guides
• Health Sciences Library

Evidence Based Nursing Practice

• Literature Review
• PICO(T) and Clinical Questions
• Study Design and Methodology
• Documenting EBP
• Additional Sources & Assistance
• MUHC Evidence Based Practice Model

Search Engines (aka Databases)

Included is CINAHL which covers both the nursing literature and the allied health fields such  as:Cardiopulmonary Technology, Emergency Services, Health Education, Med/Lab Technology, Occupational Therapy, Physical Therapy, Radiologic Technology and Respiratory Therapy. Date Coverage: 1982-present

• Cochrane Database of Systematic Reviews Full-text systematic reviews of the effects of healthcare by The Cochrane Collaboration

Free account can be created for personalization. Date Coverage: 1966-date. Maximum Users: 44 Truncation: Truncation: *; Wildcard: ? Tutorial: http://www.screencast.com/users/lottes/folders/Jing/media/a7a0252d-37e8-4b6e-8d59-29f60369ada3

Date Coverage: 1970-present; selected access back to 1823 Maximum Users: Unlimited Truncation: Truncation: *; Wildcard: ? Search Guide : http://help.scopus.com Tutorial : https://supporthub.scopus.com Auto Alerts : http://library.missouri.edu/search/databases/alertslist.htm#scopus

• Google Scholar Broad coverage of disciplines, easy to use, use when you need to search the full text of articles
• Google Books Search full text of books. Good for very specific questions or hard to find information.
• DynaMed Plus A good place to find guidelines.
• UpToDate Overviews of thousands of diseases & drugs. NOTE: access is limited by place & person. SHP & SON students will need to access UpToDate from the Health Sciences Library or the clinics.

A Review by Any Other Name

Literature reviews can range from quick and dirty to detailed and thorough.  When searching for the evidence or working under the title of 'evidence based' it's best to be as detailed and thorough as possible.

• documenting your work - where & how you searched;
• searching multiple databases;
• using multiple terms for each of your PICO concepts;
• using AND/OR (Boolean logic)- correctly of course!
• asking a librarian either for assistance or to perform the literature search for you
• << Previous: PICO(T) and Clinical Questions
• Next: Study Design and Methodology >>
• Last Updated: Jul 3, 2024 10:41 AM
• URL: https://libraryguides.missouri.edu/EBNP

• Finding Full Text
• Finding Full Text in PubMed
• Accessing Articles from Off Campus
• Evaluating Articles
• Borrowing Print Books
• Evaluating Web Sources
• Writing and Citing Sources
• Primary and Secondary Sources in the Sciences
• Careers and Test Prep
• More Resources
• Nurse Theorists
• Nursing Informatics

Evidence-Based Practice

What is evidence-based practice.

The American Nurses Association defines evidence-based practice (EBP) as:

Evidence-based practice in nursing involves providing holistic, quality care based on the most up-to-date research and knowledge rather than traditional methods, advice from colleagues, or personal beliefs.  Evidence-based nursing draws upon critical reasoning and judgment skills developed through experience and training. You can practice evidence-based nursing interventions by  following five crucial steps  that serve as guidelines for making patient care decisions. This process includes incorporating the best external evidence, your clinical expertise, and the patient's values and expectations. Ask a clear question about the patient's issue and determine an ultimate goal, such as improving a procedure to help their specific condition.  Acquire the best evidence by searching relevant clinical articles from legitimate sources. Appraise the resources gathered to determine if the information is valid, of optimal quality compared to the evidence levels, and relevant for the patient. Apply the evidence to clinical practice by making decisions based on your nursing expertise and the new information. Assess outcomes to determine if the treatment was effective and should be considered for other patients.

You are likely already practicing some tenets of EBP in your coursework and research without even knowing it.  However, there are a few things specific to EBP that you may not have explicitly discussed before.  Read on.

Developing a Clear Question

In evidence-based research, clear questions are often developed using the PICO(T) model, which helps define your research goals and form a search strategy.

PICO(T) is a mnemonic used to describe the four elements of a good clinical foreground question.

P = Population/Problem:   How would I describe the problem or a group of patients similar to mine?

I = Intervention:   What main intervention, prognostic factor or exposure am I considering?

C = Comparison:   Is there an alternative to compare with the intervention?

O = Outcome:   What do I hope to accomplish, measure, improve or affect?

T = Time (Sometimes):   Over what period of time will the intervention take place or be assessed?

EBP Tenet #2

Understanding source types.

Once you've defined your research question, your next step is to look for appropriate sources and articles.  You have experience doing this in previous courses, but something new that you may encounter are the terms filtered and unfiltered sources.  Unfiltered information encompasses the primary source articles that you may have discussed before, while filtered information encompasses the secondary sources.  In EBP, filtered information is the best source for clinical decision-making.

Here's the Evidence Pyramid, with each level of evidence explained in more detail below:

Finding Information in Nursing Databases

Now that you know what kinds of information is available, your next step is to search and find it.  Some of our regular library databases that you've used for other courses include EBP material.  These databases have some features that you may not have used before and will be helpful for this project:

• CINAHL Plus with Full Text This link opens in a new window CINAHL contains a great deal of research, including systematic reviews. Under "Limit Your Results" use the clinical queries filter and/or the evidence-based practice checkbox to limit your search to evidence-based materials. Also, under Publication Type you can limit your search to systematic reviews.
• PubMed This link opens in a new window Use the "Additional filters" button on the left to search for specific types of articles, such as Clinical Trial, Randomized Controlled Trial, or Meta-Analysis.
• Medline with Full Text This link opens in a new window Use the "Publication Type" option to search for specific types of articles, such as Clinical Trial, Randomized Controlled Trial, or Meta-Analysis.
• APA PsycArticles - list of titles
• APA PsycINFO This link opens in a new window Seach can be limited to systematic review in the methodology section. more... less... Indexing to dissertations, book chapters, technical reports and other documents.
• R2 Online Library (Nursing) This link opens in a new window For background information/expert opinion level only.

Finding Information on Websites

There are also several websites that contain EBP information that may not be available in the library databases:

• Trip Medical Database Trip is a clinical search engine designed to allow users to quickly and easily find and use high-quality research evidence to support their practice and/or care. The PICO search option makes it easy to plug in terms from your PICO(T) question.
• Agency for Healthcare Research and Quality The Agency for Healthcare Research and Quality (AHRQ) is the lead Federal agency charged with improving the safety and quality of healthcare for all Americans. Their Clinical Guidelines and Recommendations provide the basis for sound clinical practice guidelines and recommendations.
• PubMed Clinical Queries This tool uses predefined filters to help you quickly refine PubMed searches on clinical or disease-specific topics. To use this tool, enter your search terms in the search bar and select filters before searching.

For more training in EBP research, view the following tutorials from other institutions:

• Evidence-Based Practice: An Interprofessional Tutorial An open educational resource on EBP.
• Evidence Based Practice A tutorial from the Health Sciences Library at the University of North Carolina at Chapel Hill
• Evidence-based medicine (EBM) toolkit Fundamentals to learn, practise, and discuss EBM, from the BMJ Best Practice site
• << Previous: Nursing Informatics
• Last Updated: Jul 22, 2024 3:56 PM
• URL: https://stevenson.libguides.com/nursing

• Study protocol
• Open access
• Published: 15 July 2024

Implementing evidence-based practices to improve primary care for high-risk patients: study protocol for the VA high-RIsk VETerans (RIVET) type III effectiveness-implementation trial

• Elvira E. Jimenez   ORCID: orcid.org/0009-0009-7189-6370 1 , 2 , 4 ,
• Ann-Marie Rosland 3 , 4   na1 ,
• Susan E. Stockdale 1 , 2   na1 ,
• Ashok Reddy 5 , 6   na1 ,
• Michelle S. Wong 1   na1 ,
• Natasha Torrence 3 , 4   na1 ,
• Alexis Huynh 1   na1 &
• Evelyn T. Chang 1 , 7   na1  

Implementation Science Communications volume  5 , Article number:  75 ( 2024 ) Cite this article

178 Accesses

3 Altmetric

Metrics details

Patients with significant multimorbidity and other factors that make healthcare challenging to access and coordinate are at high risk for poor health outcomes. Although most (93%) of Veterans’ Health Administration (VHA) patients at high risk for hospitalization or death (“high-risk Veterans”) are primarily managed by primary care teams, few of these teams have implemented evidence-based practices (EBPs) known to improve outcomes for the high-risk patient population’s complex healthcare issues. Effective implementation strategies could increase adoption of these EBPs in primary care; however, the most effective implementation strategies to increase evidence-based care for high-risk patients are unknown.

The high-RIsk VETerans (RIVET) Quality Enhancement Research Initiative (QUERI) will compare two variants of Evidence-Based Quality Improvement (EBQI) strategies to implement two distinct EBPs for high-risk Veterans: individual coaching (EBQI-IC; tailored training with individual implementation sites to meet site-specific needs) versus learning collaborative (EBQI-LC; implementation sites trained in groups to encourage collaboration among sites). One EBP, Comprehensive Assessment and Care Planning (CACP), guides teams in addressing patients’ cognitive, functional, and social needs through a comprehensive care plan. The other EBP, Medication Adherence Assessment (MAA), addresses common challenges to medication adherence using a patient-centered approach.

We will recruit and randomize 16 sites to either EBQI-IC or EBQI-LC to implement one of the EBPs, chosen by the site. Each site will have a site champion (front-line staff) who will participate in 18 months of EBQI facilitation.

We will use a mixed-methods type 3 hybrid Effectiveness-Implementation trial to test EBQI-IC versus EBQI-LC versus usual care using a Concurrent Stepped Wedge design. We will use the Practical, Robust Implementation and Sustainability Model (PRISM) framework to compare and evaluate Reach, Effectiveness, Adoption, Implementation, and costs. We will then assess the maintenance/sustainment and spread of both EBPs in primary care after the 18-month implementation period. Our primary outcome will be Reach, measured by the percentage of eligible high-risk patients who received the EBP.

Our study will identify which implementation strategy is most effective overall, and under various contexts, accounting for unique barriers, facilitators, EBP characteristics, and adaptations. Ultimately this study will identify ways for primary care clinics and teams to choose implementation strategies that can improve care and outcomes for patients with complex healthcare needs.

Trial registration

ClinicalTrials.gov, NCT05050643. Registered September 9th, 2021, https://clinicaltrials.gov/study/NCT05050643

Protocol version

This protocol is Version 1.0 which was created on 6/3/2020.

Contributions to the literature

The first study to compare the effectiveness of Evidence-Based Quality Improvement (EBQI) conducted individually with one site or conducted with a group of sites.

Implementation of evidence-based practices to improve care for high-risk patients in primary care.

Patients who are at the highest risk for hospitalization (“high-risk patients”) are a heterogenous subset of patients who have significant multimorbidity and pose the most significant medical challenges within any healthcare organization [ 1 ]. These patients are at high risk for poor health outcomes and account for the majority of the Veterans Health Administration (VHA) healthcare costs [ 2 ], similar to other healthcare systems [ 3 , 4 ]. Previous Medicare demonstrations, such as advanced primary care home models called Comprehensive Primary Care Plus (CPC +), have shown that caring for high-risk patients can be challenging despite financial alignment that promotes coordination of care delivery [ 5 ]. Primary care teams bear most of the responsibility in caring for complex patients; in VHA over 93% of high-risk Veterans are managed by general primary care teams, despite the availability of specialized primary care teams for patients with advanced health conditions [ 6 ]. Yet, complex, high-risk patients often do not receive the most effective evidence-based care within general primary care teams [ 7 ].

Many of the evidence-based practices (EBPs) have been ineffective in the management of high-risk patients due to the lack of EBPs that properly address multimorbidity— most EBPs focus on a single problem [ 8 , 9 ]. However, there are a few EBPs that have shown to be effective for high-risk patients in geriatrics and other specialized settings, such as comprehensive assessments, individualized care plans, and care coordination among the multidisciplinary team members [ 10 , 11 , 12 ]. Evidence also supports patient-centered approaches to support self-management for high-risk patients with competing medical and self-care demands, including shared decision making and health coaching [ 12 , 13 , 14 ]. However, primary care teams have not implemented these EBPs widely [ 15 , 16 ]. Despite the availability of these effective practices, the most effective implementation strategies to increase uptake of EBPs for high-risk patients in primary care are unknown. EBQI has been used successfully to implement complex EBPs in VA primary care, such as the patient-centered medical home model [ 17 ] and primary care-mental health integration [ 18 , 19 ]. EBQI is, in fact, a bundle of implementation strategies that emphasizes a systematic approach to developing a researcher-clinical partnership that engages national, regional, and local-level senior organizational leaders and local QI teams in adapting and implementing EBPs in the context of prior evidence and local practice conditions [ 17 ]. Core elements of EBQI include engaging multi-level multidisciplinary stakeholders in evidence-based agenda setting (developing a “QI Roadmap”), training clinical champions in QI methods to meet agenda goals, and practice facilitation [ 17 , 18 , 20 , 21 ]. The theoretical basis underlying EBQI elements includes theories of organizational change [ 22 , 23 , 24 , 25 ], clinical quality improvement, [ 26 , 27 , 28 ] complex adaptive systems [ 25 ], and diffusion of innovation [ 29 ]; each element can be mapped to documented implementation strategies [ 30 ]. However, beyond these core elements, EBQI initiatives have widely varied in the extent and types of interactions with implementation facilitators, specifically in practice facilitation and training. Some initiatives have combined EBQI core components with individual ongoing consultation [ 17 , 20 ], which can require significant researcher and quality coordinator time and resources. Other VHA EBQI initiatives have used across-site learning collaboratives (i.e., depression collaborative care [ 18 , 19 ] and opioid use disorder [ 31 ]), which may require fewer resources and impact a greater number of health professionals. While both variations have been effective, individual site-level consultation has never been empirically compared with learning collaboratives; implementers lack guidance on which of these strategies are effective in what setting.

This study uses a mixed-methods type 3 hybrid implementation-effectiveness evaluation using a Concurrent Stepped Wedge design, evaluation of two separate interventions in different clusters, [ 32 ] to compare the two variants of EBQI aimed at increasing reach of the proposed EBPs (CACP or MAA). The Practical, Robust Implementation and Sustainability Model (PRISM) framework (Fig.  1 ) [ 33 , 34 ] will guide the planning, implementation, and evaluation of the RIVET Program. The PRISM framework specifies contextual factors which align well with the components of our implementation strategies, and which will guide our evaluation of factors that influence Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) outcomes [ 34 ]. The Consolidated Framework for Implementation Research (CFRI) framework will identify implementation determinants, i.e., barriers and facilitators to implementation.

RIVET implementation timeline for each site

Evidence‑based practices (EBPs)

We selected two EBPs that guide primary care teams to identify modifiable needs among high-risk patients through standardized assessments and utilize the expertise of the multidisciplinary staff in the primary care team: Comprehensive Assessment and Care Plan (CACP) and Medication Adherence Assessment (MAA).

EBP Comprehensive Assessment and Care Plan (CACP) for high-risk patients

The CACP is an assessment that helps the primary care team to develop an individualized treatment plan based on identified needs for high-risk patients of any age [ 35 ]. It was adapted from the Comprehensive Geriatric Assessment (CGA), which has been shown in multiple randomized control trials (RCTs) to lead to improved outcomes for older adults with complex care needs [ 36 , 37 ]. According to meta-analyses, the CGA has consistently led to improved outcomes for frail, older adults, such as decreased mortality (OR 0.86, 95% CI 0.75–0.98), decreased readmissions (OR 0.88, 95% CI 0.79–0.98), and decreased length of stay (1.63–40.7 days in intervention group vs 1.8–42.8 days in usual care) compared to those who did not receive the CGA [ 38 , 39 , 40 ].

While the CGA assesses several domains that are important for complex patients, some domains may not be broadly applicable to high-risk Veterans of all ages (i.e., nutrition, vision, hearing, continence). According to our analyses, half of high-risk Veterans are younger than 65 years old and have greater psychiatric comorbidities than older high-risk Veterans [ 6 ]. We added domains to specifically assess for modifiable risk factors that are common among high-risk Veterans (e.g., transportation assistance, health literacy, behavioral health symptoms, and coordination with non-VHA healthcare systems) [ 41 , 42 , 43 ]. The CACP screening questions are taken from standard sources, including the National Academy of Medicine Recommendations for High-Need Patients [ 44 ] and the Protocol for Responding to and Assessing Patients’ Assets, Risks, and Experiences (PRAPARE) [ 45 ].

The CACP prompts the primary care team member to explore identified risk factors further or to refer to another team member for an in-depth assessment if needed. After the assessment, the CACP is then used to guide the development and implementation of an individualized treatment plan that addresses the patient’s health-related needs in the context of the patient’s preferences [ 10 ]. The treatment plan can be coordinated and monitored through team huddles (consisting of the primary care provider, nurses, clerk) or monthly interdisciplinary team meetings.

EBP Medication Adherence Assessment (MAA)

High-risk Veterans often experience complicated medication regimens, potential side effects, and other known barriers to medication adherence [ 46 ]. Medication nonadherence represents a common problem among multimorbid patients [ 47 ], and is one of the largest contributors to preventable emergency department visits and hospitalizations among high-risk Veterans. Many approaches to improving medication adherence are limited by a focus on particular diseases rather than the entire medication regimen and by a focus on the healthcare providers’ point of view, rather than on patients’ goals and agency around medication taking [ 48 ].

A standardized Medication Adherence Assessment (MAA) guides primary care teams to assess for barriers and challenges to medication adherence through open-ended questions and enables primary care teams to understand high-risk patients’ goals and preferences and better impact medication-taking behaviors across the patients’ medication conditions [ 12 , 13 ]. The MAA prompts the clinician to employ specific strategies around adherence and potentially use motivational interviewing or health coaching if the patient is ambivalent about taking a medication. Health coaching has emerged as an effective, patient-centered, collaborative approach to understand patient goals and preferences and enhance patients’ adherence to modifiable health behaviors [ 49 ]. Health coaching is a type of cognitive-based behavior change technique that employs motivational interviewing and goal-setting to guide the patient to change health behaviors, such as diet, exercise, and medication adherence [ 49 ]. Meta-analyses have shown that cognitive-based behavior change techniques (e.g., health coaching) to improve medication management are associated with an effect size of 0.34 (95% CI 0.23–0.46) on improved medication adherence [ 50 ].

EBQI Implementation strategies

The high-Risk VETeran (RIVET) Program will compare two variants of Evidence-Based Quality Improvement (EBQI)—practice facilitation through Individual Consultation (IC) or through Learning Collaboratives (LC) to determine which of the two is the most effective and less costly implementation strategy and explore which is better suited for which contexts.

Individual Consultation (IC)

Individual consultation, often described as coaching or supervision, is endorsed by implementation experts as an effective implementation strategy in and of itself. In RIVET, external facilitators from the study team will provide training to QI participants, who are front-line primary care staff, from an individual medical facility. The RIVET IC will provide regular individualized EBQI training and coaching to implement the EBP. Consultation allows experts to tailor complex skills and training to needs of the organization and to the QI participants, using active learning and providing practice opportunities [ 51 ]. It also provides QI participants with problem-solving skills and accountability [ 51 ]. Literature has demonstrated increased uptake and adherence to EBPs and increase sustainability with IC [ 51 , 52 ].

Learning Collaboratives (LC)

Learning collaboratives are also widely used in healthcare settings and are an effective implementation strategy [ 30 ]. The RIVET learning collaboratives consists of external facilitators from the study team providing quality improvement training to QI participants from multiple sites in a group, and encourages interaction and collaboration among QI participants (e.g., providing feedback to each other). The effectiveness of learning collaboratives vary, but generally, they have demonstrated improvement in health professionals’ knowledge, problem-solving skills and attitude, and teamwork [ 53 ]. The mechanisms by which learning collaboratives may be effective include factors within an organization and factors between multiple organizations. In terms of factors within an organization, participation in learning collaboratives may increase staff confidence in using data to make decisions and to problem solve, increase accountability by making standards explicit, promote peer reflection, and facilitate teamwork, shared responsibility, and joint problem solving [ 53 ]. Mechanistic factors between organizations include normative pressure from peers, friendly competition, a platform for capacity building, and collaboration with other sites [ 53 ].

Site selection and eligibility

We will implement the EBPs at 16 primary care sites, targeting those with high ambulatory care-sensitive hospitalization rates. Each site will be implementing a single EBP. Implementation strategies are randomized by site. Site implementation consists of four overlapping cohorts of four sites (three randomized to LC and one to IC); all will undergo 18-months of RIVET facilitation (see Figure#1). Time periods without active implementation will serve as the usual care periods for EBQI strategies. Usual care sites will receive Office of Primary Care educational campaigns and dissemination of tools for high-risk patients among primary care teams. VHA regional leaders or VHA facility leaders will select the EBP for primary care sites to implement.

VHA primary care uses a multi-disciplinary patient-centered team-based approach (Patient-Aligned Care Teams; PACTs) where teams of health care professionals provide longitudinal care to a panel of patients [ 35 ]. Team members include a primary care provider, nurses, clerk, integrated mental health provider, social worker, and a pharmacist. Teams have access to multiple dashboards and reports for care management, including the Care Assessment Need (CAN) score [ 54 ], which describes the patient’s risk for future VA hospitalization or death by percentile.

EBQI activities

We first engaged regional and local multidisciplinary stakeholders to discuss implementation of each EBP, such as the target patient population and the clinical staff who might perform the assessment, developing a “QI Roadmap.” QI training and practice facilitation spans 18 months of video calls with site clinical champions, selected by their facility leadership. Meetings are led by a trained RIVET external facilitator and include structured QI didactics, designing Plan-Do-Study-Act cycles, coaching, review of data, and developing structured action plans (Fig.  1 ). The RIVET implementation team provides quarterly data reports and regularly discuss next steps with the champions.

The same structured QI didactics are utilized for both LC and IC groups (Fig.  1 ). The IC sites participate in individual meetings with the clinical champions every month, on average. Learning collaboratives consist of three EBQI-LC sites that participate in monthly meetings. All sites randomly assigned to EBQI-LC participate in the same learning collaborative regardless of EBP (CACP vs MAA), as both EBPs involve the same goal, the same target population, in the same clinical setting, similar to prior initiatives [ 17 , 31 , 55 ].

Data sources

Data sources include VHA Central Data Warehouse (CDW) administrative data, surveys to high-risk patients, surveys to Primary Care staff, key stakeholder interviews, implementation facilitation logs, time activity surveys, and site administrative documents.

EHR Administrative data

VHA CDW contains data on patient characteristics, outpatient encounters, provider types for each encounter, acute and inpatient care utilization, medication fill history, and Healthcare Effectiveness Data and Information Set (HEDIS) quality metric status. Health factor administrative data is generated by templates constructed for each EBP within the EHR. Managerial Cost Accounting (MCA) data will be utilized for the cost analyses.

High-risk patient surveys

Patient surveys will collect data on patient experiences among high-risk primary care patients at participating sites. Patient surveys will be mailed at the beginning and end of the 18-month active implementation phase to 500 randomly selected high-risk patients empaneled to primary care teams at each site, sampled cross-sectionally at each time period with replacement. Patient eligibility is based on the following criteria: a Care Assessment Need (CAN) score ≥ 90th percentile within the month prior to the sampling date; a visit with primary care within the last six months of the sampling date; and empanelment to the clinical champion team’s panel. When possible, patient experience and satisfaction questions were sourced from the Patient-Centered Medical Home (PCMH) version of the VHA Survey of Healthcare Experiences of Patients (SHEP), based on the Consumer Assessment of Healthcare Providers and Systems (CAHPS). Additional survey items measure direct impacts of specific EBPs (such as medication adherence), and items that may impact patient engagement in and benefit from EBPs, such as trust in their primary care provider (PCP). See Table  1 for details on included measures.

Clinical staff surveys

Clinical staff surveys will be administered site-wide, to assess factors that may influence update of RIVET EBPs at the clinic, 1) other tools, resources and practices used when managing high-risk patients, and 2) exposure to RIVET EBPs (post-implementation only), and 3) confidence with practices promoted by RIVET EBPs and with overall high-risk patient care. Most items were derived from previous VA primary care staff surveys, including those conducted for the purposes of evaluating staff experiences and approaches to high-risk patient care [ 65 ]. Electronic surveys will be sent to all primary care providers, nurses, and medical assistants on eligible teams at participating sites at the beginning and end of the 18-month active implementation phase. Eligible teams include general and women’s health primary care teams, as well as any ‘specialty’ primary care teams (e.g., geriatric primary care) that the site’s champion considers eligible for EBP spread. Clinician eligibility criteria includes being a member of at least one PACT teamlet at the RIVET site; being either a physician, physician assistant, nurse practitioner, registered nurse, licensed practical or vocational nurse, a medical assistant, or a health technician; providing direct patient care at the site; and working at the primary care site for at least three months.

Key stakeholder interviews

Guided by the Practical, Robust Implementation and Sustainability Model (PRISM) and Consolidated Framework for Implementation Research (CFIR) frameworks, we will conduct pre- and post- semi-structured qualitative telephone interviews with key middle managers (Primary Care Medical Director, Primary Care Nursing lead, Social Work lead, Integrated Mental Health lead, Pharmacy lead) and frontline clinicians who participate in EBQI and implementation activities. The interviews will assess readiness (inner context) and its subconstructs of leadership and engagement, available resources, and access to knowledge and information; implementation climate (inner context) and its subconstructs of relative priority and values; implementation process and its subconstructs of engaging key stakeholders and executing the implementation plan; characteristics of Individuals and its subconstruct of knowledge and beliefs about the intervention; and intervention characteristics and its subconstructs of relative advantage and complexity.

Implementation facilitation logs

We will use templated implementation facilitation logs to collect information about participants’ attendance at EBQI activities (including facilitation sessions and other meetings), participants’ role in RIVET, plan-do-study-act cycles, ‘real time’ adaptations to the EBPs, and barriers to implementation. The Implementation facilitators and coordinator will complete the implementation facilitation logs for both EBQI-IC and EBQI-LC sites after each meeting and any contact with implementation sites and leadership.

Time activity surveys

We will administer weekly time surveys to the RIVET implementation team which will capture RIVET staff time spent in various implementation activities. To assess time spent on RIVET implementation activities by site participants outside of facilitation sessions and other meetings with the RIVET implementation team, we will conduct brief monthly polls via Teams during facilitation sessions.

Periodic reflections

Using a semi-structured interview guide, a 30-min recorded monthly meeting will be conducted with the facilitation staff to elicit information and their overall impression of implementation progress and process. The meeting will document any implementation challenges and successes, adaptations, stakeholder engagement and relevant contextual issues. We will assess contextual factors, such as organizational readiness (leadership engagement, resources, access to knowledge and information) and anticipated barriers/facilitators at each site.

LC and IC site administrative documents

The site administrative documents include the QI roadmap for each EBP, site action plans developed by clinical champions, meeting minutes, written reports and presentations to leadership, and attendance records.

The Practical, Robust Implementation and Sustainability Model (PRISM) framework) [ 33 , 34 ] will guide the planning, implementation, and evaluation of the RIVET Program. The PRISM framework specifies contextual factors which align well with the components of our implementation strategies, and which will guide our evaluation of factors that influence Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) outcomes [ 34 ]. We will use mixed methods to evaluate RE-AIM outcomes, EBP fidelity, implementation strategy fidelity, adaptations, costs, benefits, and value.

Reach is defined in this study as the proportion of high-risk patients on each study team’s panel that received one of the EBPs during the 18-month implementation period. See Table  2 for details on included reach measures. We define “high-risk patients” using a VHA-specific risk score called the Care Assessment Need score, previously developed and validated by VA through machine-learning to predict a patient’s risk for a VHA hospitalization or mortality [ 54 ]. We define ‘receipt’ as having the EBP assessment at least partially documented in the electronic health record (EHR). We will also examine the patient characteristics (sociodemographic, Elixhauser comorbidity score [ 66 ]) of eligible patients who did vs. did not receive the EBP.

Effectiveness

EBP effectiveness measures will include type of referrals or actions generated by each EBP, and whether they were completed, as measured by EHR template use, administrative consult data, and encounters. We will also use administrative EHR data to measure number of patient encounters with primary care team members (social work, pharmacist, nurse, integrated mental health), as an indicator of engagement of primary care teams for high-risk patient care. Impacts of RIVET on patient experience will be evaluated with a patient survey that includes measures (described above) of satisfaction and access to resources and support for caring for high-risk patients. We will also assess clinical performance metrics that are expected to be directly impacted by each EBP. For both EBPs, we will measure acute care utilization, such as emergency department (ED) visits, total acute and ambulatory care-sensitive (ACS)-related hospitalizations. For MAA, we will also measure adherence to medications for common chronic conditions, such as hypertension and diabetes. See Table  2 for details on included effectiveness measures.

We will assess adoption by measuring number and proportion of staff trained on each EBP, and how many and which types of staff delivered each EBP (representativeness), using administrative training records and administrative clinical data for each EBP.

Implementation fidelity

We will assess implementation strategy fidelity using the EBQI fidelity assessment tool [ 55 ]. The fidelity assessment tool draws from data collected from key stakeholder interviews, implementation facilitation logs, administrative documents, and weekly time diaries (described below). We will apply criteria to rate sites as high, medium, or low fidelity on the EBQI elements. Using the implementation facilitation logs, we will assess participation in the EBQI activities by frontline providers, staff, and leadership.

Maintenance

We assess maintenance by the extent to which EBPs are implemented after practice facilitation ends (e.g., about 18 months). We also plan to study if the EBPs are spread to other primary care teams within the site and to other sites within the healthcare system.

Outcomes and data analyses

To analyze our primary outcome, receipt of each EBP among top 10th percentile high-risk patients during the 18-month implementation period (Reach), we will model uptake of both practices in our Concurrent Stepped Wedge Design as a multilevel hierarchical model with a repeated cross-sectional data structure in which sites are followed over time. In this design, the data structure includes patients at level 1 (where Reach, the main outcome of interest, in measured), nested within time at level 2, and nested within sites at level 3. The main predictors will be site implementation strategy assignment (EBQI-IC vs. EBQI-LC vs Usual Care) over time (measure as the number of quarter). We will first describe differences in trends using unadjusted analyses, then add covariates for patient characteristics (i.e., age, sex, Elixhauser comorbidity score [ 66 ]) and site-level covariates (i.e., number of unique patients served in primary care, rural vs urban). Secondary analyses of Reach will include models by key patient subgroups, including those hospitalized in the 90 days prior to the quarter examined, and models with an interaction term for assigned implementation strategy by EBQI fidelity level (as defined above), to examine how implementation strategy fidelity may have impacted Reach.

To analyze our secondary outcomes of care processes, patient experiences, provider experiences, and performance measures (Effectiveness), we will first model unadjusted trends in outcomes over time for each EBP, by site implementation strategy assignment. We will then model outcomes using two or three-level (depending on whether the metric is measured at the patient or site level) hierarchical models based on the concurrent Stepped Wedge Design using repeated cross-sectional data, adjusted for the same covariates included in the models for our primary outcome. In models of medication adherence, we will add covariates for medication regimen complexity, as indicated by total number of classes of medication prescribed for the condition of interest, and patient co-pay status. If we find differences in outcomes, we will perform mediation analyses that consider how Reach and EBP fidelity may have mediated outcomes [ 67 ].

To assess differences in adoption between EBQI-IC and EBQI-LC sites, we will use bivariate analyses to compare the number and proportion of staff trained on each EBP, and how many and which types of staff delivered each EBP. To assess differences in implementation strategy fidelity among sites, we will compare the number of sites with high, medium, and low fidelity on each element of EBQI, as well as overall fidelity to EBQI. Similarly, we will compare number of sites with overall high, medium, or low EBP fidelity among EBQI-IC vs EBQI-LC sites.

We will qualitatively assess the impact of contextual factors on implementation, using a matrix analysis approach [ 68 ] to explore a priori themes based on the interview guides, but also allow for emergent themes. Specifically, we will code and analyze interview data for core elements of the EBQI implementation strategy and for contextual features, intervention characteristics, and implementation infrastructure guided by the PRISM framework [ 33 ]. Two trained qualitative analysts will construct and validate the codebook [ 69 ]. Using this codebook, one analyst will code all interviews, and a second qualitative analyst will review all coding. After generating a report for all codes, they will use a matrix analysis approach [ 68 ] to populate a participant-by-theme matrix and create site level summaries for each theme to facilitate comparisons between EBQI-IC and EBQI-LC sites. To ensure rigor, summaries will also be reviewed, compared with the original data, and discussed by at least two analysts to reach consensus for any discrepancies. Finally, we will link this qualitative matrix with site-level implementation strategy and EBP fidelity measures to compare how specific contextual features, intervention characteristics, and implementation infrastructure impacted fidelity.

Return-On-Investment (ROI) Analyses

We will conduct a budget impact analysis to inform the sustainability of each EBP. Following the VHA recommendation to evaluate cost of projects [ 70 ], we will identify the relevant costs associated with implementation of the EBP, the EBP itself, and the consequences of the EBP (e.g., healthcare utilization). Using a micro-costing approach [ 71 ], we will collect costs for EBQI-IC and EBQI-LC sites, measuring: 1) implementation costs as one-time costs to develop the intervention; and 2) intervention and downstream costs as costs that would be incurred by facilities adopting the EBPs (e.g., site participants; RIVET implementation team members’ time spent in training, meetings and preparing for meetings; staff time performing the intervention). To capture RIVET implementation team staff time spent in various implementation activities, data will be collected through the implementation facilitation log, administrative documents, and weekly time surveys. For both EBPs, clinical staff will document the estimated time spent to complete EBP through an EHR template. Finally, we will identify the costs of healthcare utilization which may be impacted by the implementation of these EBPs from the VA administrative data, such as change in outpatient utilization (e.g., primary care, social work, mental health, pharmacy) and inpatient utilization. This will be done by estimating the excess cost for patients exposed to the EBP compared to a control group of unexposed patients using multivariable regression models to control for measured confounders. We will use generalized linear models for continuous data and two-part models for semi-continuous data with distributional families chosen to best fit the data. If preliminary data reveals that medication adherence changed with MAA implementation, we will also include changes in pharmacy costs.

We will measure whether EBPs were maintained during each sites’ sustainment period (time period following the 18-month active implementation period), how EBPs spread within the original sites and to new sites, and what factors are associated with maintenance and spread. We will also assess adaptations made to the EBPs in response to changing VHA context.

For sustainment and spread, we will continue to assess receipt of each EBP among top 10% CAN score patients (Reach) longitudinally during the sustainment period by measuring 6 and 12 months after the active implementation phase. We will incorporate Adoption measures to assess spread within the implementation site and in new sites by measuring how many additional staff were trained on each EBP, proportion of staff trained on each EBP, and which types of healthcare staff delivered each EBP (representativeness). We will compare EBQI-IC and EBQI-LC sites to determine which implementation strategy is most effective for sustaining the EBPs. Using the matrix analysis approach described above, we will analyze qualitative interview data to explore the role of contextual factors on sustainment, and the “fit” between context, intervention, and implementation strategy on sustainability.

The RIVET project aims to implement two evidence-based assessments to improve the management of the high-risk patient population within VHA primary care using two EBQI strategies. This work will add to a much needed body of literature evaluating the effectiveness of different approaches to EBQI to implement EBPs within primary care [ 1 ]. It is the first study to compare the effectiveness of EBQI conducted with individual sites (EBQI-IC) vs conducted with groups of sites as a learning collaborative (EBQI-LC) and to compare which implementation strategy is most effective under various contexts accounting for unique barriers, facilitators, and adaptations. Additionally, RIVET will provide evidence on which of the two strategies is the most cost-efficient strategy. Comparing EBQI-LC and EBQI-IC will allow our VHA primary care leaders to tailor the implementation strategy to the primary care context in preparation for widespread implementation. While our project focuses on EBPs for high-risk patients, we anticipate that this comparison of EBQI strategies can inform those implementing EBPs with other primary care patient populations.

The EBPs aim to systematically identify modifiable risk factors within primary care for patients with complex needs—enabling primary care to provide comprehensive and holistic care. Furthermore, by selecting the most effective, less burdensome, and less costly implementation strategy ensures greater buy-in from clinical leadership interested in offering advanced primary care and frontline staff who are often overwhelmed with clinical demands and chronic staffing shortages.

We anticipate several potential challenges to optimal implementation. The major challenge for the RIVET project is that, as with any project embedded in pragmatic healthcare system operations, it is vulnerable to national and local VHA contextual factors. Specifically, success of the project can be compromised by VHA staffing changes within the sites and study teams. In addition, since primary care teams are tasked with a wide variety of care, new health system initiatives and external circumstances (e.g., pandemic-induced changes in care delivery) can unexpectedly compete with high-risk patient care priorities. In addition, the active implementation period requires 18-month engagement from clinical champions to learn EBQI practices and to properly use the EBP in their routine care of high-risk patients. Finally, RIVET EBQI IC and LC sessions will be held virtually. Although most provider training has moved to virtual modalities post-Covid, the best methods to keep staff engaged may vary over time and setting. The RIVET project will not only implement EBP tools that will help better manage high-risk Veterans at 16 VA sites but will provide the tools and evidence on the best implementation strategies for primary care staff at VHA working to improve high-risk patient care and primary care delivery.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Abbreviations

Ambulatory care-sensitive

Comprehensive Assessment and Care Planning

Consumer Assessment of Healthcare Providers and Systems

Care Assessment Need

Consolidated Framework for Implementation Research

Comprehensive Geriatric Assessment

Comprehensive Primary Care Plus

Central Data Warehouse

Evidence-based practice

Evidence-Based Quality Improvement

Emergency department

Electronic health record

Healthcare Effectiveness Data and Information Set

Individual coaching

Learning collaborative

Medication Adherence Assessment

Managerial Cost Accounting

Office of Primary Care

Patient-Aligned Care Team

Patient-Centered Medical Home

Primary care provider

Protocol for Responding to and Assessing Patients’ Assets, Risks, and Experiences

Practical, Robust Implementation and Sustainability Model

Quality Enhancement Research Initiative

Randomized control trial

Reach, Effectiveness, Adoption, Implementation, and Maintenance

High-RIsk VETerans

Return-On-Investment

Survey of Healthcare Experiences of Patients

Veterans’ Health Administration

Hempel S, et al. Evidence-based quality improvement: a scoping review of the literature. J Gen Intern Med. 2022;37(16):4257–67.

Article   PubMed   PubMed Central   Google Scholar  

Zulman DM, et al. Multimorbidity and healthcare utilisation among high-cost patients in the US Veterans Affairs Health Care System. BMJ Open. 2015;5(4):e007771.

Marengoni A, et al. Aging with multimorbidity: a systematic review of the literature. Ageing Res Rev. 2011;10(4):430–9.

Article   PubMed   Google Scholar  

Smith SM, et al. Managing patients with multimorbidity: systematic review of interventions in primary care and community settings. BMJ. 2012;345:e5205.

Higgins TC, O’Malley AS, Keith RE. Exploring and overcoming the challenges primary care practices face with care management of high-risk patients in CPC+: a mixed-methods study. J Gen Intern Med. 2021;36(10):3008–14.

Chang ET, et al. Use of general primary care, specialized primary care, and other veterans affairs services among high-risk veterans. JAMA Netw Open. 2020;3(6):e208120.

Hebert PL, et al. Patient-centered medical home initiative produced modest economic results for Veterans Health Administration, 2010–12. Health Aff (Millwood). 2014;33(6):980–7.

Skou ST, et al. Multimorbidity. Nat Rev Dis Primers. 2022;8(1):48.

Boyd CM, et al. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. JAMA. 2005;294(6):716–24.

Article   CAS   PubMed   Google Scholar  

Boult C, Wieland GD. Comprehensive primary care for older patients with multiple chronic conditions: “Nobody rushes you through.” JAMA. 2010;304(17):1936–43.

O’Toole TP, et al. Tailoring Outreach efforts to increase primary care use among homeless veterans: results of a randomized controlled trial. J Gen Intern Med. 2015;30:886–98.

Poitras M-E, et al. What are the effective elements in patient-centered and multimorbidity care? A scoping review. BMC Health Serv Res. 2018;18(1):446.

Tinetti M, et al. Patient priority-directed decision making and care for older adults with multiple chronic conditions. Clin Geriatr Med. 2016;32(2):261.

Jones A, et al. A national evaluation of homeless and nonhomeless veterans’ experiences with primary care. Psychol Serv. 2017;14(2):174.

Midboe AM, et al. Implementing motivational interviewing in primary care: the role of provider characteristics. Transl Behav Med. 2011;1(4):588–94.

Stockdale S, et al. What do patient-centered medical home (PCMH) teams need to better manage care for their patients at high-risk for hospitalization or mortality? In: in society of general internal medicine annual meeting. Washington, DC: Society of General Internal Medicine; 2019.

Google Scholar  

Rubenstein LV, et al. Evidence-based quality improvement: a method for helping managed primary care practices become patient centered. J Gen Intern Med. 2014;29:589–97.

Article   PubMed Central   Google Scholar  

Rubenstein LV, et al. Using evidence-based quality improvement methods for translating depression collaborative care research into practice. Fam Syst Health. 2010;28:91–113.

Yano EM, et al. Cluster randomized trial of a multilevel evidence-based quality improvement approach to tailoring VA patient aligned care teams to the needs of women Veterans. Implement Sci. 2016;11(1):101.

Hamilton A, et al. Engaging multilevel stakeholders in an implementation trial of evidence-based quality improvement in va women’s health primary care. Transl Behav Med. 2017;7(3):478.

Lukas CV, et al. Transformational change in health care systems: an organizational model. Health Care Manage Rev. 2007;32(4):309–20.

Ovretveit J. Improvement leaders: what do they and should they do? A summary of a review of research. Qual Saf Health Care. 2010;19(6):490–2.

PubMed   Google Scholar  

Øvretveit J. Understanding the conditions for improvement: research to discover which context influences affect improvement success. BMJ Qual Saf. 2011;20 Suppl 1(Suppl_1):i18–23.

Crabtree BF, et al. Primary care practice transformation is hard work: insights from a 15-year developmental program of research. Med Care. 2011;49 Suppl(l):28–35.

Article   Google Scholar  

Kaplan HC, et al. The Model for Understanding Success in Quality (MUSIQ): building a theory of context in healthcare quality improvement. BMJ Qual Saf. 2012;21(1):13–20.

King G, et al. A framework of operating models for interdisciplinary research programs in clinical service organizations. Eval Program Plann. 2008;31(2):160–73.

Berwick DM. The science of improvement. JAMA. 2008;299(10):1182–4.

Greenhalgh T, et al. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82(4):581–629.

Powell BJ, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.

Chang ET, Oberman RS, Cohen AN, Taylor SL, Gumm E, Mardian AS, et al. Increasing access to medications for opioid use disorder and complementary and integrative health services in primary care. J Gen Intern Med. 2020;35:918-26.

Lyons VH, Li L, Hughes JP, Rowhani-Rahbar A. Proposed variations of the stepped-wedge design can be used to accommodate multiple interventions. J Clin Epidemiol. 2017;86:160–7.

McCreight MS, et al. Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Transl Behav Med. 2019;9(6):1002–11.

Glasgow RE, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019;7:64.

Elsawy B, Higgins KE. The geriatric assessment. Am Fam Physician. 2011;83(1):48–56.

Reuben DB, et al. A randomized trial of comprehensive geriatric assessment in the care of hospitalized patients. N Engl J Med. 1995;332(20):1345–50.

Garrard JW, et al. Comprehensive geriatric assessment in primary care: a systematic review . Aging Clin Exp Res, 2019;32(2):197–205.

Ellis G, et al. Comprehensive geriatric assessment for older adults admitted to hospital: meta-analysis of randomised controlled trials. BMJ. 2011;343:d6553.

Ellis G, et al. Comprehensive geriatric assessment for older adults admitted to hospital. Cochrane Database Syst Rev. 2017;9(9):006211.

Stuck AE, et al. Comprehensive geriatric assessment: a meta-analysis of controlled trials. Lancet. 1993;342(8878):1032–6.

Hatef E, et al. Integrating social and behavioral determinants of health into patient care and population health at Veterans Health Administration: a conceptual framework and an assessment of available individual and population level data sources and evidence-based measurements. AIMS Public Health. 2019;6(3):209–24.

Zullig LL, et al. A systematic review of conceptual frameworks of medical complexity and new model development. J Gen Intern Med. 2016;31(3):329–37.

Cornell PY, et al. Embedding social workers in veterans health administration primary care teams reduces emergency department visits. Health Aff (Millwood). 2020;39:603. https://doi.org/10.1377/hlthaff.2019.01589 .

Long P, et al. Effective care for high-need patients: opportunities for improving outcomes, values, and health. In: in NAM special publication leadership consortium for a value & science-driven health system. Washington, DC: National Academy of Medicine; 2017.

National Association of Community Health Centers. PRAPARE: Protocol for Responding to and Assessing Patients' Assets, Risks, and Experiences . 2019; Research and Data:[Available from: http://www.nachc.org/research-and-data/prapare/ . Cited 2019 Dec 9

Hanlon J, et al. Suboptimal prescribing in older inpatients and outpatients. J Am Geriatr Soc. 2001;49(2):200.

Osterberg L, Blaschke T. Adherence to medication. N Engl J Med. 2005;353(5):487–97.

Viswanathan M, et al. Medication therapy management interventions in outpatient settings: a systematic review and meta-analysis. JAMA Intern Med. 2015;175(1):76–87.

Gierisch J, Hughes J, Edelman D. The Effectiveness of Health Coaching. Washington (DC): Department of Veterans Affairs (US); 2017. Available from: https://www.ncbi.nlm.nih.gov/books/NBK487700/ .

Easthall C, Song F, Bhattacharya D. A meta-analysis of cognitive-based behaviour change techniques as interventions to improve medication adherence. BMJ Open. 2013;3(8):e002749.

Nadeem E, Gleacher A, Beidas RS. Consultation as an implementation strategy for evidence-based practices across multiple contexts: unpacking the black box. Adm Policy Ment Health. 2013;40(6):439–50.

Waltz TJ, et al. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implement Sci. 2019;14(1):42.

Zamboni K, et al. How and under what circumstances do quality improvement collaboratives lead to better outcomes? A systematic review. Implement Sci. 2020;15(1):27.

Wang L, et al. Predicting risk of hospitalization or death among patients receiving primary care in the Veterans Health Administration. Med Care. 2013;51(4):368–73.

Stockdale SE, et al. Assessing fidelity to evidence-based quality improvement as an implementation strategy for patient-centered medical home transformation in the Veterans Health Administration. Implement Sci. 2020;15(1):18.

Department of Veterans Affairs. SHEP Patient Centered Medical Homes (PCMH) Short Form (VA 10-1465-5). Department of Veterans Affairs. 2021. https://omb.report/icr/202102-2900-017/doc/109373600 . Accessed 7 Nov 2024.

Voils CI, et al. Initial validation of a self-report measure of the extent of and reasons for medication nonadherence. Med Care. 2012;50(12):1013–9.

Zulman DM, et al. Effects of intensive primary care on high-need patient experiences: survey findings from a veterans affairs randomized quality improvement trial. J Gen Intern Med. 2019;34(Suppl 1):75–81.

Rosland AM, et al. Effectiveness of a health coaching intervention for patient-family dyads to improve outcomes among adults with diabetes: a randomized clinical trial. JAMA Netw Open. 2022;5(11):e2237960.

Jensen RE, et al. Validation of the PROMIS physical function measures in a diverse US population-based cohort of cancer patients. Qual Life Res. 2015;24(10):2333–44.

Chew LD, et al. Validation of screening questions for limited health literacy in a large VA outpatient population. J Gen Intern Med. 2008;23(5):561–6.

Billioux A, Verlander K, Anthony S, Alley D. Standardized Screening for Health-Related Social Needs in Clinical Settings: The Accountable Health Communities Screening Tool. NAM Perspectives. Discussion Paper. National Academy of Medicine; 2017. https://nam.edu/wpcontent/uploads/2017/05/Standardized-Screening-for-Health-Related-Social-Needs-in-Clinical-Settings.pdf . Accessed 10 July 2024.

Anderson GO, Thayer CE. Loneliness and social connections: A national survey of adults 45 and older. Washington, DC: AARP Foundation; 2018.

Deverts DJ, et al. Comparing the effectiveness of Family Support for Health Action (FAM-ACT) with traditional community health worker-led interventions to improve adult diabetes management and outcomes: study protocol for a randomized controlled trial. Trials. 2022;23(1):841.

Meredith LS, et al. Can using an intensive management program improve primary care staff experiences with caring for high-risk patients? Fed Pract. 2021;38(2):68.

PubMed   PubMed Central   Google Scholar  

Elixhauser A, et al. Comorbidity measures for use with administrative data. Med Care. 1998;36(1):8–27.

Baron R, Kenny D. The moderator-mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol. 1986;51(6):1173.

Averill JB. Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qual Health Res. 2002;12(6):855–66.

Miles M, Huberman A. Qualitative data analysis: an expanded sourcebook. Thousand Oaks, CA: Sage; 1994.

Veteran's Administration Health Economics Resource Center. What are the steps to conduct economic analyses for implementation research? 2019 August 7, 2019; Available from: https://www.herc.research.va.gov/include/page.asp?id=implementation-steps . Cited 2019 December 3

Liu CF, et al. Organizational cost of quality improvement for depression care. Health Serv Res. 2009;44(1):225–44.

Download references

Acknowledgements

We would like to acknowledge Kelsey Cummings, MS and Emily Wong, MPH for their contribution to the implementation of the project; Bridget Kranke, MSSA, LSW for assisting with editing; and the VA Office of Primary Care (OPC), Geriatrics and Extended Care (GEC), Patient-Centered Care and Cultural Transformation (OPCC&CT), Primary Care Improvement and Innovation and the primary care leads from Veterans Integrated Services Networks (VISNs) 9, 10, 12 for their advice and support.

The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs, or the US government, or other affiliated institutions.

This project is funded by the VA Quality Enhancement Research Initiative (QUERI) program, project #: QUE-20–018.

Author information

Ann-Marie Rosland, Susan Stockdale, Ashok Reddy, Michelle Wong, Natasha Torrence, Alexis Huynh and Evelyn Chang contributed equally to this work.

Authors and Affiliations

Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), VA Greater Los Angeles Healthcare System, 11301 Wilshire Blvd, Los Angeles, CA, 90073, USA

Elvira E. Jimenez, Susan E. Stockdale, Michelle S. Wong, Alexis Huynh & Evelyn T. Chang

Department of Neurology, David Geffen School of Medicine, University of California Los Angeles (UCLA), 760 Westwood Plaza, Los Angeles, CA, 90095, USA

Elvira E. Jimenez & Susan E. Stockdale

Center for Health Equity Research and Promotion (CHERP), VA Pittsburgh Healthcare System, 1 University Dr, Pittsburgh, PA, 15240, USA

Ann-Marie Rosland & Natasha Torrence

Caring for Complex Chronic Conditions Research Center & Division of General Internal Medicine, School of Medicine, University of Pittsburgh, 3550 Terrace St, Pittsburgh, PA, 15213, USA

Elvira E. Jimenez, Ann-Marie Rosland & Natasha Torrence

Department of Medicine, Division of General Internal Medicine, Harborview Medical Center, University of Washington, 325 Ninth Ave, Box 359780, Seattle, WA, 98104, USA

Ashok Reddy

Center for Veteran-Centered and Value-Driven Care, VA Puget Sound Health Care System, 1660 S Columbian Way, Seattle, WA, 98108, USA

Division of General Internal Medicine, Department of Medicine, David Geffen School of Medicine, UCLA, 740 Charles E Young Dr S, Los Angeles, CA, 90095, USA

Evelyn T. Chang

You can also search for this author in PubMed   Google Scholar

Contributions

Contributions to the creation and design of the funded project includes AMR, SS, and EC. Development of the project design at implementation stage includes EJ, AMR, SS, AR, MW, NT, AH, and EC. All listed authors contributed to writing/editing the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Elvira E. Jimenez .

Ethics declarations

Ethics approval and consent to participate.

The RIVET project was determined to qualify as non-research conducted under the authority of VHA Office of Primary Care, as it was designed and implemented with the intent to improve internal patient care at VHA and not conduct systematic research to advance scientific knowledge base. In accordance with VHA policies and guidelines, this program is considered as non-research by IRB (Subcommittee on Human Studies) of the VHA Greater Los Angeles Healthcare System Research Service (691/151) which is authorized to determined projects as non-research activities for which additional IRB oversight is not required, as defined per VHA Handbook 1058.05 in the section “Officials Authorized to Provide Documentation of VHA Program Office Non-Research Operations Activities” and later updated in Sect. 5a of the VHA Program Guide 1200.21.

Consent for publication

Not applicable.

Competing interests

Author EC has a consultancy with Behavioral Health Services, Inc. The remaining authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Jimenez, E.E., Rosland, AM., Stockdale, S.E. et al. Implementing evidence-based practices to improve primary care for high-risk patients: study protocol for the VA high-RIsk VETerans (RIVET) type III effectiveness-implementation trial. Implement Sci Commun 5 , 75 (2024). https://doi.org/10.1186/s43058-024-00613-9

Download citation

Received : 30 May 2024

Accepted : 08 July 2024

Published : 15 July 2024

DOI : https://doi.org/10.1186/s43058-024-00613-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Primary Health Care
• Multimorbidity
• Medication Adherence
• Needs Assessment

Implementation Science Communications

ISSN: 2662-2211

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]