literature review sources in research methodology

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Literature Review | Guide, Examples, & Templates

How to Write a Literature Review | Guide, Examples, & Templates

Published on January 2, 2023 by Shona McCombes . Revised on September 11, 2023.

What is a literature review? A literature review is a survey of scholarly sources on a specific topic. It provides an overview of current knowledge, allowing you to identify relevant theories, methods, and gaps in the existing research that you can later apply to your paper, thesis, or dissertation topic .

There are five key steps to writing a literature review:

• Search for relevant literature
• Evaluate sources
• Identify themes, debates, and gaps
• Outline the structure
• Write your literature review

A good literature review doesn’t just summarize sources—it analyzes, synthesizes , and critically evaluates to give a clear picture of the state of knowledge on the subject.

Table of contents

What is the purpose of a literature review, examples of literature reviews, step 1 – search for relevant literature, step 2 – evaluate and select sources, step 3 – identify themes, debates, and gaps, step 4 – outline your literature review’s structure, step 5 – write your literature review, free lecture slides, other interesting articles, frequently asked questions, introduction.

• Quick Run-through
• Step 1 & 2

When you write a thesis , dissertation , or research paper , you will likely have to conduct a literature review to situate your research within existing knowledge. The literature review gives you a chance to:

• Demonstrate your familiarity with the topic and its scholarly context
• Develop a theoretical framework and methodology for your research
• Position your work in relation to other researchers and theorists
• Show how your research addresses a gap or contributes to a debate
• Evaluate the current state of research and demonstrate your knowledge of the scholarly debates around your topic.

Writing literature reviews is a particularly important skill if you want to apply for graduate school or pursue a career in research. We’ve written a step-by-step guide that you can follow below.

A faster, more affordable way to improve your paper

Scribbr’s new AI Proofreader checks your document and corrects spelling, grammar, and punctuation mistakes with near-human accuracy and the efficiency of AI!

Proofread my paper

Writing literature reviews can be quite challenging! A good starting point could be to look at some examples, depending on what kind of literature review you’d like to write.

• Example literature review #1: “Why Do People Migrate? A Review of the Theoretical Literature” ( Theoretical literature review about the development of economic migration theory from the 1950s to today.)
• Example literature review #2: “Literature review as a research methodology: An overview and guidelines” ( Methodological literature review about interdisciplinary knowledge acquisition and production.)
• Example literature review #3: “The Use of Technology in English Language Learning: A Literature Review” ( Thematic literature review about the effects of technology on language acquisition.)
• Example literature review #4: “Learners’ Listening Comprehension Difficulties in English Language Learning: A Literature Review” ( Chronological literature review about how the concept of listening skills has changed over time.)

You can also check out our templates with literature review examples and sample outlines at the links below.

Download Word doc Download Google doc

Before you begin searching for literature, you need a clearly defined topic .

If you are writing the literature review section of a dissertation or research paper, you will search for literature related to your research problem and questions .

Make a list of keywords

Start by creating a list of keywords related to your research question. Include each of the key concepts or variables you’re interested in, and list any synonyms and related terms. You can add to this list as you discover new keywords in the process of your literature search.

• Social media, Facebook, Instagram, Twitter, Snapchat, TikTok
• Body image, self-perception, self-esteem, mental health
• Generation Z, teenagers, adolescents, youth

Search for relevant sources

Use your keywords to begin searching for sources. Some useful databases to search for journals and articles include:

• Your university’s library catalogue
• Google Scholar
• Project Muse (humanities and social sciences)
• Medline (life sciences and biomedicine)
• EconLit (economics)
• Inspec (physics, engineering and computer science)

You can also use boolean operators to help narrow down your search.

Make sure to read the abstract to find out whether an article is relevant to your question. When you find a useful book or article, you can check the bibliography to find other relevant sources.

You likely won’t be able to read absolutely everything that has been written on your topic, so it will be necessary to evaluate which sources are most relevant to your research question.

For each publication, ask yourself:

• What question or problem is the author addressing?
• What are the key concepts and how are they defined?
• What are the key theories, models, and methods?
• Does the research use established frameworks or take an innovative approach?
• What are the results and conclusions of the study?
• How does the publication relate to other literature in the field? Does it confirm, add to, or challenge established knowledge?
• What are the strengths and weaknesses of the research?

Make sure the sources you use are credible , and make sure you read any landmark studies and major theories in your field of research.

You can use our template to summarize and evaluate sources you’re thinking about using. Click on either button below to download.

Take notes and cite your sources

As you read, you should also begin the writing process. Take notes that you can later incorporate into the text of your literature review.

It is important to keep track of your sources with citations to avoid plagiarism . It can be helpful to make an annotated bibliography , where you compile full citation information and write a paragraph of summary and analysis for each source. This helps you remember what you read and saves time later in the process.

To begin organizing your literature review’s argument and structure, be sure you understand the connections and relationships between the sources you’ve read. Based on your reading and notes, you can look for:

• Trends and patterns (in theory, method or results): do certain approaches become more or less popular over time?
• Themes: what questions or concepts recur across the literature?
• Debates, conflicts and contradictions: where do sources disagree?
• Pivotal publications: are there any influential theories or studies that changed the direction of the field?
• Gaps: what is missing from the literature? Are there weaknesses that need to be addressed?

This step will help you work out the structure of your literature review and (if applicable) show how your own research will contribute to existing knowledge.

• Most research has focused on young women.
• There is an increasing interest in the visual aspects of social media.
• But there is still a lack of robust research on highly visual platforms like Instagram and Snapchat—this is a gap that you could address in your own research.

There are various approaches to organizing the body of a literature review. Depending on the length of your literature review, you can combine several of these strategies (for example, your overall structure might be thematic, but each theme is discussed chronologically).

Chronological

The simplest approach is to trace the development of the topic over time. However, if you choose this strategy, be careful to avoid simply listing and summarizing sources in order.

Try to analyze patterns, turning points and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred.

If you have found some recurring central themes, you can organize your literature review into subsections that address different aspects of the topic.

For example, if you are reviewing literature about inequalities in migrant health outcomes, key themes might include healthcare policy, language barriers, cultural attitudes, legal status, and economic access.

Methodological

If you draw your sources from different disciplines or fields that use a variety of research methods , you might want to compare the results and conclusions that emerge from different approaches. For example:

• Look at what results have emerged in qualitative versus quantitative research
• Discuss how the topic has been approached by empirical versus theoretical scholarship
• Divide the literature into sociological, historical, and cultural sources

Theoretical

A literature review is often the foundation for a theoretical framework . You can use it to discuss various theories, models, and definitions of key concepts.

You might argue for the relevance of a specific theoretical approach, or combine various theoretical concepts to create a framework for your research.

Like any other academic text , your literature review should have an introduction , a main body, and a conclusion . What you include in each depends on the objective of your literature review.

The introduction should clearly establish the focus and purpose of the literature review.

Depending on the length of your literature review, you might want to divide the body into subsections. You can use a subheading for each theme, time period, or methodological approach.

As you write, you can follow these tips:

• Summarize and synthesize: give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: don’t just paraphrase other researchers — add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically evaluate: mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: use transition words and topic sentences to draw connections, comparisons and contrasts

In the conclusion, you should summarize the key findings you have taken from the literature and emphasize their significance.

When you’ve finished writing and revising your literature review, don’t forget to proofread thoroughly before submitting. Not a language expert? Check out Scribbr’s professional proofreading services !

This article has been adapted into lecture slides that you can use to teach your students about writing a literature review.

Scribbr slides are free to use, customize, and distribute for educational purposes.

Open Google Slides Download PowerPoint

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarize yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

The literature review usually comes near the beginning of your thesis or dissertation . After the introduction , it grounds your research in a scholarly field and leads directly to your theoretical framework or methodology .

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, September 11). How to Write a Literature Review | Guide, Examples, & Templates. Scribbr. Retrieved November 14, 2023, from https://www.scribbr.com/dissertation/literature-review/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research methodology | steps & tips, how to write a research proposal | examples & templates, what is your plagiarism score.

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Library subject guides

Literature reviews.

• Introduction
• Plan your search
• Where to search
• Refine and update your search
• Finding grey literature
• Writing the review
• Referencing

Research methods overview

Finding literature on research methodologies, sage research methods online.

• Get material not at RMIT
• Further help

What are research methods?

Research methodology is the specific strategies, processes, or techniques utilised in the collection of information that is created and analysed.

The methodology section of a research paper, or thesis, enables the reader to critically evaluate the study’s validity and reliability by addressing how the data was collected or generated, and how it was analysed.

Types of research methods

There are three main types of research methods which use different designs for data collection.  

(1) Qualitative research

Qualitative research gathers data about lived experiences, emotions or behaviours, and the meanings individuals attach to them. It assists in enabling researchers to gain a better understanding of complex concepts, social interactions or cultural phenomena. This type of research is useful in the exploration of how or why things have occurred, interpreting events and describing actions.

Examples of qualitative research designs include:

• focus groups
• observations
• document analysis
• oral history or life stories  

(2) Quantitative research

Quantitative research gathers numerical data which can be ranked, measured or categorised through statistical analysis. It assists with uncovering patterns or relationships, and for making generalisations. This type of research is useful for finding out how many, how much, how often, or to what extent.

Examples of quantitative research designs include:

• surveys or questionnaires
• observation
• document screening
• experiments  

(3) Mixed method research

Mixed Methods research integrates both Qualitative research and Quantitative research. It provides a holistic approach combining and analysing the statistical data with deeper contextualised insights. Using Mixed Methods also enables triangulation, or verification, of the data from two or more sources.

Sometimes in your literature review, you might need to discuss and evaluate relevant research methodologies in order to justify your own choice of research methodology.

When searching for literature on research methodologies it is important to search across a range of sources. No single information source will supply all that you need. Selecting appropriate sources will depend upon your research topic.

Developing a robust search strategy will help reduce irrelevant results. It is good practice to plan a strategy before you start to search.

Search tips

(1) free text keywords.

Free text searching is the use of natural language words to conduct your search. Use selective free text keywords such as: phenomenological, "lived experience", "grounded theory", "life experiences", "focus groups", interview, quantitative, survey, validity, variance, correlation and statistical.

To locate books on your desired methodology, try LibrarySearch . Remember to use  refine  options such as books, ebooks, subject, and publication date.  

(2) Subject headings in Databases

Databases categorise their records using subject terms, or a controlled vocabulary (thesaurus). These subject headings may be useful to use, in addition to utilising free text keywords in a database search.

Subject headings will differ across databases, for example, the PubMed database uses 'Qualitative Research' whilst the CINHAL database uses 'Qualitative Studies.'  

(3) Limiting search results

Databases enable sets of results to be limited or filtered by specific fields, look for options such as Publication Type, Article Type, etc. and apply them to your search.  

(4) Browse the Library shelves

To find books on  research methods  browse the Library shelves at call number  001.42

• SAGE Research Methods Online SAGE Research Methods Online (SRMO) is a research tool supported by a newly devised taxonomy that links content and methods terms. It provides the most comprehensive picture available today of research methods (quantitative, qualitative and mixed methods) across the social and behavioural sciences.

SAGE Research Methods Overview  (2:07 min) by SAGE Publishing  ( YouTube ) 

• << Previous: Referencing
• Next: Get material not at RMIT >>
• Last Updated: Nov 17, 2023 4:49 PM
• URL: https://rmit.libguides.com/literature-review

Research Methods

• Getting Started
• Literature Review Research
• Research Design
• Research Design By Discipline
• SAGE Research Methods
• Teaching with SAGE Research Methods

Literature Review

• What is a Literature Review?
• What is NOT a Literature Review?
• Purposes of a Literature Review
• Types of Literature Reviews
• Literature Reviews vs. Systematic Reviews
• Systematic vs. Meta-Analysis

Literature Review  is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

• Summarizes and analyzes previous research relevant to a topic
• Includes scholarly books and articles published in academic journals
• Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

• Help gather ideas or information
• Keep up to date in current trends and findings
• Help develop new questions

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas.
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic.
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
• Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University 

Keep in mind the following, a literature review is NOT:

Not an essay 

Not an annotated bibliography  in which you summarize each article that you have reviewed.  A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper   where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

• provides thorough knowledge of previous studies; introduces seminal works.
• helps focus one’s own research topic.
• identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
• suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
• identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
• helps the researcher avoid repetition of earlier research.
• suggests unexplored populations.
• determines whether past studies agree or disagree; identifies controversy in the literature.
• tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature."  Educational Researcher  36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015),  Literature reviews vs systematic reviews.  Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Systematic review or meta-analysis?

A  systematic review  answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A  meta-analysis  is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

• clearly stated objectives with pre-defined eligibility criteria for studies
• explicit, reproducible methodology
• a systematic search that attempts to identify all studies
• assessment of the validity of the findings of the included studies (e.g. risk of bias)
• systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis. 

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review.  More information on meta-analyses can be found in  Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies.  It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings.  Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted.  In that case, an integrative review is an appropriate strategy. 

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

• << Previous: Getting Started
• Next: Research Design >>
• Last Updated: Aug 21, 2023 4:07 PM
• URL: https://guides.lib.udel.edu/researchmethods

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.

Handbook of eHealth Evaluation: An Evidence-based Approach [Internet].

Chapter 9 methods for literature reviews.

Guy Paré and Spyros Kitsiou .

9.1. Introduction

Literature reviews play a critical role in scholarship because science remains, first and foremost, a cumulative endeavour ( vom Brocke et al., 2009 ). As in any academic discipline, rigorous knowledge syntheses are becoming indispensable in keeping up with an exponentially growing eHealth literature, assisting practitioners, academics, and graduate students in finding, evaluating, and synthesizing the contents of many empirical and conceptual papers. Among other methods, literature reviews are essential for: (a) identifying what has been written on a subject or topic; (b) determining the extent to which a specific research area reveals any interpretable trends or patterns; (c) aggregating empirical findings related to a narrow research question to support evidence-based practice; (d) generating new frameworks and theories; and (e) identifying topics or questions requiring more investigation ( Paré, Trudel, Jaana, & Kitsiou, 2015 ).

Literature reviews can take two major forms. The most prevalent one is the “literature review” or “background” section within a journal paper or a chapter in a graduate thesis. This section synthesizes the extant literature and usually identifies the gaps in knowledge that the empirical study addresses ( Sylvester, Tate, & Johnstone, 2013 ). It may also provide a theoretical foundation for the proposed study, substantiate the presence of the research problem, justify the research as one that contributes something new to the cumulated knowledge, or validate the methods and approaches for the proposed study ( Hart, 1998 ; Levy & Ellis, 2006 ).

The second form of literature review, which is the focus of this chapter, constitutes an original and valuable work of research in and of itself ( Paré et al., 2015 ). Rather than providing a base for a researcher’s own work, it creates a solid starting point for all members of the community interested in a particular area or topic ( Mulrow, 1987 ). The so-called “review article” is a journal-length paper which has an overarching purpose to synthesize the literature in a field, without collecting or analyzing any primary data ( Green, Johnson, & Adams, 2006 ).

When appropriately conducted, review articles represent powerful information sources for practitioners looking for state-of-the art evidence to guide their decision-making and work practices ( Paré et al., 2015 ). Further, high-quality reviews become frequently cited pieces of work which researchers seek out as a first clear outline of the literature when undertaking empirical studies ( Cooper, 1988 ; Rowe, 2014 ). Scholars who track and gauge the impact of articles have found that review papers are cited and downloaded more often than any other type of published article ( Cronin, Ryan, & Coughlan, 2008 ; Montori, Wilczynski, Morgan, Haynes, & Hedges, 2003 ; Patsopoulos, Analatos, & Ioannidis, 2005 ). The reason for their popularity may be the fact that reading the review enables one to have an overview, if not a detailed knowledge of the area in question, as well as references to the most useful primary sources ( Cronin et al., 2008 ). Although they are not easy to conduct, the commitment to complete a review article provides a tremendous service to one’s academic community ( Paré et al., 2015 ; Petticrew & Roberts, 2006 ). Most, if not all, peer-reviewed journals in the fields of medical informatics publish review articles of some type.

The main objectives of this chapter are fourfold: (a) to provide an overview of the major steps and activities involved in conducting a stand-alone literature review; (b) to describe and contrast the different types of review articles that can contribute to the eHealth knowledge base; (c) to illustrate each review type with one or two examples from the eHealth literature; and (d) to provide a series of recommendations for prospective authors of review articles in this domain.

9.2. Overview of the Literature Review Process and Steps

As explained in Templier and Paré (2015) , there are six generic steps involved in conducting a review article:

• formulating the research question(s) and objective(s),
• searching the extant literature,
• screening for inclusion,
• assessing the quality of primary studies,
• extracting data, and
• analyzing data.

Although these steps are presented here in sequential order, one must keep in mind that the review process can be iterative and that many activities can be initiated during the planning stage and later refined during subsequent phases ( Finfgeld-Connett & Johnson, 2013 ; Kitchenham & Charters, 2007 ).

Formulating the research question(s) and objective(s): As a first step, members of the review team must appropriately justify the need for the review itself ( Petticrew & Roberts, 2006 ), identify the review’s main objective(s) ( Okoli & Schabram, 2010 ), and define the concepts or variables at the heart of their synthesis ( Cooper & Hedges, 2009 ; Webster & Watson, 2002 ). Importantly, they also need to articulate the research question(s) they propose to investigate ( Kitchenham & Charters, 2007 ). In this regard, we concur with Jesson, Matheson, and Lacey (2011) that clearly articulated research questions are key ingredients that guide the entire review methodology; they underscore the type of information that is needed, inform the search for and selection of relevant literature, and guide or orient the subsequent analysis. Searching the extant literature: The next step consists of searching the literature and making decisions about the suitability of material to be considered in the review ( Cooper, 1988 ). There exist three main coverage strategies. First, exhaustive coverage means an effort is made to be as comprehensive as possible in order to ensure that all relevant studies, published and unpublished, are included in the review and, thus, conclusions are based on this all-inclusive knowledge base. The second type of coverage consists of presenting materials that are representative of most other works in a given field or area. Often authors who adopt this strategy will search for relevant articles in a small number of top-tier journals in a field ( Paré et al., 2015 ). In the third strategy, the review team concentrates on prior works that have been central or pivotal to a particular topic. This may include empirical studies or conceptual papers that initiated a line of investigation, changed how problems or questions were framed, introduced new methods or concepts, or engendered important debate ( Cooper, 1988 ). Screening for inclusion: The following step consists of evaluating the applicability of the material identified in the preceding step ( Levy & Ellis, 2006 ; vom Brocke et al., 2009 ). Once a group of potential studies has been identified, members of the review team must screen them to determine their relevance ( Petticrew & Roberts, 2006 ). A set of predetermined rules provides a basis for including or excluding certain studies. This exercise requires a significant investment on the part of researchers, who must ensure enhanced objectivity and avoid biases or mistakes. As discussed later in this chapter, for certain types of reviews there must be at least two independent reviewers involved in the screening process and a procedure to resolve disagreements must also be in place ( Liberati et al., 2009 ; Shea et al., 2009 ). Assessing the quality of primary studies: In addition to screening material for inclusion, members of the review team may need to assess the scientific quality of the selected studies, that is, appraise the rigour of the research design and methods. Such formal assessment, which is usually conducted independently by at least two coders, helps members of the review team refine which studies to include in the final sample, determine whether or not the differences in quality may affect their conclusions, or guide how they analyze the data and interpret the findings ( Petticrew & Roberts, 2006 ). Ascribing quality scores to each primary study or considering through domain-based evaluations which study components have or have not been designed and executed appropriately makes it possible to reflect on the extent to which the selected study addresses possible biases and maximizes validity ( Shea et al., 2009 ). Extracting data: The following step involves gathering or extracting applicable information from each primary study included in the sample and deciding what is relevant to the problem of interest ( Cooper & Hedges, 2009 ). Indeed, the type of data that should be recorded mainly depends on the initial research questions ( Okoli & Schabram, 2010 ). However, important information may also be gathered about how, when, where and by whom the primary study was conducted, the research design and methods, or qualitative/quantitative results ( Cooper & Hedges, 2009 ). Analyzing and synthesizing data : As a final step, members of the review team must collate, summarize, aggregate, organize, and compare the evidence extracted from the included studies. The extracted data must be presented in a meaningful way that suggests a new contribution to the extant literature ( Jesson et al., 2011 ). Webster and Watson (2002) warn researchers that literature reviews should be much more than lists of papers and should provide a coherent lens to make sense of extant knowledge on a given topic. There exist several methods and techniques for synthesizing quantitative (e.g., frequency analysis, meta-analysis) and qualitative (e.g., grounded theory, narrative analysis, meta-ethnography) evidence ( Dixon-Woods, Agarwal, Jones, Young, & Sutton, 2005 ; Thomas & Harden, 2008 ).

9.3. Types of Review Articles and Brief Illustrations

EHealth researchers have at their disposal a number of approaches and methods for making sense out of existing literature, all with the purpose of casting current research findings into historical contexts or explaining contradictions that might exist among a set of primary research studies conducted on a particular topic. Our classification scheme is largely inspired from Paré and colleagues’ (2015) typology. Below we present and illustrate those review types that we feel are central to the growth and development of the eHealth domain.

9.3.1. Narrative Reviews

The narrative review is the “traditional” way of reviewing the extant literature and is skewed towards a qualitative interpretation of prior knowledge ( Sylvester et al., 2013 ). Put simply, a narrative review attempts to summarize or synthesize what has been written on a particular topic but does not seek generalization or cumulative knowledge from what is reviewed ( Davies, 2000 ; Green et al., 2006 ). Instead, the review team often undertakes the task of accumulating and synthesizing the literature to demonstrate the value of a particular point of view ( Baumeister & Leary, 1997 ). As such, reviewers may selectively ignore or limit the attention paid to certain studies in order to make a point. In this rather unsystematic approach, the selection of information from primary articles is subjective, lacks explicit criteria for inclusion and can lead to biased interpretations or inferences ( Green et al., 2006 ). There are several narrative reviews in the particular eHealth domain, as in all fields, which follow such an unstructured approach ( Silva et al., 2015 ; Paul et al., 2015 ).

Despite these criticisms, this type of review can be very useful in gathering together a volume of literature in a specific subject area and synthesizing it. As mentioned above, its primary purpose is to provide the reader with a comprehensive background for understanding current knowledge and highlighting the significance of new research ( Cronin et al., 2008 ). Faculty like to use narrative reviews in the classroom because they are often more up to date than textbooks, provide a single source for students to reference, and expose students to peer-reviewed literature ( Green et al., 2006 ). For researchers, narrative reviews can inspire research ideas by identifying gaps or inconsistencies in a body of knowledge, thus helping researchers to determine research questions or formulate hypotheses. Importantly, narrative reviews can also be used as educational articles to bring practitioners up to date with certain topics of issues ( Green et al., 2006 ).

Recently, there have been several efforts to introduce more rigour in narrative reviews that will elucidate common pitfalls and bring changes into their publication standards. Information systems researchers, among others, have contributed to advancing knowledge on how to structure a “traditional” review. For instance, Levy and Ellis (2006) proposed a generic framework for conducting such reviews. Their model follows the systematic data processing approach comprised of three steps, namely: (a) literature search and screening; (b) data extraction and analysis; and (c) writing the literature review. They provide detailed and very helpful instructions on how to conduct each step of the review process. As another methodological contribution, vom Brocke et al. (2009) offered a series of guidelines for conducting literature reviews, with a particular focus on how to search and extract the relevant body of knowledge. Last, Bandara, Miskon, and Fielt (2011) proposed a structured, predefined and tool-supported method to identify primary studies within a feasible scope, extract relevant content from identified articles, synthesize and analyze the findings, and effectively write and present the results of the literature review. We highly recommend that prospective authors of narrative reviews consult these useful sources before embarking on their work.

Darlow and Wen (2015) provide a good example of a highly structured narrative review in the eHealth field. These authors synthesized published articles that describe the development process of mobile health ( m-health ) interventions for patients’ cancer care self-management. As in most narrative reviews, the scope of the research questions being investigated is broad: (a) how development of these systems are carried out; (b) which methods are used to investigate these systems; and (c) what conclusions can be drawn as a result of the development of these systems. To provide clear answers to these questions, a literature search was conducted on six electronic databases and Google Scholar . The search was performed using several terms and free text words, combining them in an appropriate manner. Four inclusion and three exclusion criteria were utilized during the screening process. Both authors independently reviewed each of the identified articles to determine eligibility and extract study information. A flow diagram shows the number of studies identified, screened, and included or excluded at each stage of study selection. In terms of contributions, this review provides a series of practical recommendations for m-health intervention development.

9.3.2. Descriptive or Mapping Reviews

The primary goal of a descriptive review is to determine the extent to which a body of knowledge in a particular research topic reveals any interpretable pattern or trend with respect to pre-existing propositions, theories, methodologies or findings ( King & He, 2005 ; Paré et al., 2015 ). In contrast with narrative reviews, descriptive reviews follow a systematic and transparent procedure, including searching, screening and classifying studies ( Petersen, Vakkalanka, & Kuzniarz, 2015 ). Indeed, structured search methods are used to form a representative sample of a larger group of published works ( Paré et al., 2015 ). Further, authors of descriptive reviews extract from each study certain characteristics of interest, such as publication year, research methods, data collection techniques, and direction or strength of research outcomes (e.g., positive, negative, or non-significant) in the form of frequency analysis to produce quantitative results ( Sylvester et al., 2013 ). In essence, each study included in a descriptive review is treated as the unit of analysis and the published literature as a whole provides a database from which the authors attempt to identify any interpretable trends or draw overall conclusions about the merits of existing conceptualizations, propositions, methods or findings ( Paré et al., 2015 ). In doing so, a descriptive review may claim that its findings represent the state of the art in a particular domain ( King & He, 2005 ).

In the fields of health sciences and medical informatics, reviews that focus on examining the range, nature and evolution of a topic area are described by Anderson, Allen, Peckham, and Goodwin (2008) as mapping reviews . Like descriptive reviews, the research questions are generic and usually relate to publication patterns and trends. There is no preconceived plan to systematically review all of the literature although this can be done. Instead, researchers often present studies that are representative of most works published in a particular area and they consider a specific time frame to be mapped.

An example of this approach in the eHealth domain is offered by DeShazo, Lavallie, and Wolf (2009). The purpose of this descriptive or mapping review was to characterize publication trends in the medical informatics literature over a 20-year period (1987 to 2006). To achieve this ambitious objective, the authors performed a bibliometric analysis of medical informatics citations indexed in medline using publication trends, journal frequencies, impact factors, Medical Subject Headings (MeSH) term frequencies, and characteristics of citations. Findings revealed that there were over 77,000 medical informatics articles published during the covered period in numerous journals and that the average annual growth rate was 12%. The MeSH term analysis also suggested a strong interdisciplinary trend. Finally, average impact scores increased over time with two notable growth periods. Overall, patterns in research outputs that seem to characterize the historic trends and current components of the field of medical informatics suggest it may be a maturing discipline (DeShazo et al., 2009).

9.3.3. Scoping Reviews

Scoping reviews attempt to provide an initial indication of the potential size and nature of the extant literature on an emergent topic (Arksey & O’Malley, 2005; Daudt, van Mossel, & Scott, 2013 ; Levac, Colquhoun, & O’Brien, 2010). A scoping review may be conducted to examine the extent, range and nature of research activities in a particular area, determine the value of undertaking a full systematic review (discussed next), or identify research gaps in the extant literature ( Paré et al., 2015 ). In line with their main objective, scoping reviews usually conclude with the presentation of a detailed research agenda for future works along with potential implications for both practice and research.

Unlike narrative and descriptive reviews, the whole point of scoping the field is to be as comprehensive as possible, including grey literature (Arksey & O’Malley, 2005). Inclusion and exclusion criteria must be established to help researchers eliminate studies that are not aligned with the research questions. It is also recommended that at least two independent coders review abstracts yielded from the search strategy and then the full articles for study selection ( Daudt et al., 2013 ). The synthesized evidence from content or thematic analysis is relatively easy to present in tabular form (Arksey & O’Malley, 2005; Thomas & Harden, 2008 ).

One of the most highly cited scoping reviews in the eHealth domain was published by Archer, Fevrier-Thomas, Lokker, McKibbon, and Straus (2011) . These authors reviewed the existing literature on personal health record ( phr ) systems including design, functionality, implementation, applications, outcomes, and benefits. Seven databases were searched from 1985 to March 2010. Several search terms relating to phr s were used during this process. Two authors independently screened titles and abstracts to determine inclusion status. A second screen of full-text articles, again by two independent members of the research team, ensured that the studies described phr s. All in all, 130 articles met the criteria and their data were extracted manually into a database. The authors concluded that although there is a large amount of survey, observational, cohort/panel, and anecdotal evidence of phr benefits and satisfaction for patients, more research is needed to evaluate the results of phr implementations. Their in-depth analysis of the literature signalled that there is little solid evidence from randomized controlled trials or other studies through the use of phr s. Hence, they suggested that more research is needed that addresses the current lack of understanding of optimal functionality and usability of these systems, and how they can play a beneficial role in supporting patient self-management ( Archer et al., 2011 ).

9.3.4. Forms of Aggregative Reviews

Healthcare providers, practitioners, and policy-makers are nowadays overwhelmed with large volumes of information, including research-based evidence from numerous clinical trials and evaluation studies, assessing the effectiveness of health information technologies and interventions ( Ammenwerth & de Keizer, 2004 ; Deshazo et al., 2009 ). It is unrealistic to expect that all these disparate actors will have the time, skills, and necessary resources to identify the available evidence in the area of their expertise and consider it when making decisions. Systematic reviews that involve the rigorous application of scientific strategies aimed at limiting subjectivity and bias (i.e., systematic and random errors) can respond to this challenge.

Systematic reviews attempt to aggregate, appraise, and synthesize in a single source all empirical evidence that meet a set of previously specified eligibility criteria in order to answer a clearly formulated and often narrow research question on a particular topic of interest to support evidence-based practice ( Liberati et al., 2009 ). They adhere closely to explicit scientific principles ( Liberati et al., 2009 ) and rigorous methodological guidelines (Higgins & Green, 2008) aimed at reducing random and systematic errors that can lead to deviations from the truth in results or inferences. The use of explicit methods allows systematic reviews to aggregate a large body of research evidence, assess whether effects or relationships are in the same direction and of the same general magnitude, explain possible inconsistencies between study results, and determine the strength of the overall evidence for every outcome of interest based on the quality of included studies and the general consistency among them ( Cook, Mulrow, & Haynes, 1997 ). The main procedures of a systematic review involve:

• Formulating a review question and developing a search strategy based on explicit inclusion criteria for the identification of eligible studies (usually described in the context of a detailed review protocol).
• Searching for eligible studies using multiple databases and information sources, including grey literature sources, without any language restrictions.
• Selecting studies, extracting data, and assessing risk of bias in a duplicate manner using two independent reviewers to avoid random or systematic errors in the process.
• Analyzing data using quantitative or qualitative methods.
• Presenting results in summary of findings tables.
• Interpreting results and drawing conclusions.

Many systematic reviews, but not all, use statistical methods to combine the results of independent studies into a single quantitative estimate or summary effect size. Known as meta-analyses , these reviews use specific data extraction and statistical techniques (e.g., network, frequentist, or Bayesian meta-analyses) to calculate from each study by outcome of interest an effect size along with a confidence interval that reflects the degree of uncertainty behind the point estimate of effect ( Borenstein, Hedges, Higgins, & Rothstein, 2009 ; Deeks, Higgins, & Altman, 2008 ). Subsequently, they use fixed or random-effects analysis models to combine the results of the included studies, assess statistical heterogeneity, and calculate a weighted average of the effect estimates from the different studies, taking into account their sample sizes. The summary effect size is a value that reflects the average magnitude of the intervention effect for a particular outcome of interest or, more generally, the strength of a relationship between two variables across all studies included in the systematic review. By statistically combining data from multiple studies, meta-analyses can create more precise and reliable estimates of intervention effects than those derived from individual studies alone, when these are examined independently as discrete sources of information.

The review by Gurol-Urganci, de Jongh, Vodopivec-Jamsek, Atun, and Car (2013) on the effects of mobile phone messaging reminders for attendance at healthcare appointments is an illustrative example of a high-quality systematic review with meta-analysis. Missed appointments are a major cause of inefficiency in healthcare delivery with substantial monetary costs to health systems. These authors sought to assess whether mobile phone-based appointment reminders delivered through Short Message Service ( sms ) or Multimedia Messaging Service ( mms ) are effective in improving rates of patient attendance and reducing overall costs. To this end, they conducted a comprehensive search on multiple databases using highly sensitive search strategies without language or publication-type restrictions to identify all rct s that are eligible for inclusion. In order to minimize the risk of omitting eligible studies not captured by the original search, they supplemented all electronic searches with manual screening of trial registers and references contained in the included studies. Study selection, data extraction, and risk of bias assessments were performed inde­­pen­dently by two coders using standardized methods to ensure consistency and to eliminate potential errors. Findings from eight rct s involving 6,615 participants were pooled into meta-analyses to calculate the magnitude of effects that mobile text message reminders have on the rate of attendance at healthcare appointments compared to no reminders and phone call reminders.

Meta-analyses are regarded as powerful tools for deriving meaningful conclusions. However, there are situations in which it is neither reasonable nor appropriate to pool studies together using meta-analytic methods simply because there is extensive clinical heterogeneity between the included studies or variation in measurement tools, comparisons, or outcomes of interest. In these cases, systematic reviews can use qualitative synthesis methods such as vote counting, content analysis, classification schemes and tabulations, as an alternative approach to narratively synthesize the results of the independent studies included in the review. This form of review is known as qualitative systematic review.

A rigorous example of one such review in the eHealth domain is presented by Mickan, Atherton, Roberts, Heneghan, and Tilson (2014) on the use of handheld computers by healthcare professionals and their impact on access to information and clinical decision-making. In line with the methodological guide­lines for systematic reviews, these authors: (a) developed and registered with prospero ( www.crd.york.ac.uk/ prospero / ) an a priori review protocol; (b) conducted comprehensive searches for eligible studies using multiple databases and other supplementary strategies (e.g., forward searches); and (c) subsequently carried out study selection, data extraction, and risk of bias assessments in a duplicate manner to eliminate potential errors in the review process. Heterogeneity between the included studies in terms of reported outcomes and measures precluded the use of meta-analytic methods. To this end, the authors resorted to using narrative analysis and synthesis to describe the effectiveness of handheld computers on accessing information for clinical knowledge, adherence to safety and clinical quality guidelines, and diagnostic decision-making.

In recent years, the number of systematic reviews in the field of health informatics has increased considerably. Systematic reviews with discordant findings can cause great confusion and make it difficult for decision-makers to interpret the review-level evidence ( Moher, 2013 ). Therefore, there is a growing need for appraisal and synthesis of prior systematic reviews to ensure that decision-making is constantly informed by the best available accumulated evidence. Umbrella reviews , also known as overviews of systematic reviews, are tertiary types of evidence synthesis that aim to accomplish this; that is, they aim to compare and contrast findings from multiple systematic reviews and meta-analyses ( Becker & Oxman, 2008 ). Umbrella reviews generally adhere to the same principles and rigorous methodological guidelines used in systematic reviews. However, the unit of analysis in umbrella reviews is the systematic review rather than the primary study ( Becker & Oxman, 2008 ). Unlike systematic reviews that have a narrow focus of inquiry, umbrella reviews focus on broader research topics for which there are several potential interventions ( Smith, Devane, Begley, & Clarke, 2011 ). A recent umbrella review on the effects of home telemonitoring interventions for patients with heart failure critically appraised, compared, and synthesized evidence from 15 systematic reviews to investigate which types of home telemonitoring technologies and forms of interventions are more effective in reducing mortality and hospital admissions ( Kitsiou, Paré, & Jaana, 2015 ).

9.3.5. Realist Reviews

Realist reviews are theory-driven interpretative reviews developed to inform, enhance, or supplement conventional systematic reviews by making sense of heterogeneous evidence about complex interventions applied in diverse contexts in a way that informs policy decision-making ( Greenhalgh, Wong, Westhorp, & Pawson, 2011 ). They originated from criticisms of positivist systematic reviews which centre on their “simplistic” underlying assumptions ( Oates, 2011 ). As explained above, systematic reviews seek to identify causation. Such logic is appropriate for fields like medicine and education where findings of randomized controlled trials can be aggregated to see whether a new treatment or intervention does improve outcomes. However, many argue that it is not possible to establish such direct causal links between interventions and outcomes in fields such as social policy, management, and information systems where for any intervention there is unlikely to be a regular or consistent outcome ( Oates, 2011 ; Pawson, 2006 ; Rousseau, Manning, & Denyer, 2008 ).

To circumvent these limitations, Pawson, Greenhalgh, Harvey, and Walshe (2005) have proposed a new approach for synthesizing knowledge that seeks to unpack the mechanism of how “complex interventions” work in particular contexts. The basic research question — what works? — which is usually associated with systematic reviews changes to: what is it about this intervention that works, for whom, in what circumstances, in what respects and why? Realist reviews have no particular preference for either quantitative or qualitative evidence. As a theory-building approach, a realist review usually starts by articulating likely underlying mechanisms and then scrutinizes available evidence to find out whether and where these mechanisms are applicable ( Shepperd et al., 2009 ). Primary studies found in the extant literature are viewed as case studies which can test and modify the initial theories ( Rousseau et al., 2008 ).

The main objective pursued in the realist review conducted by Otte-Trojel, de Bont, Rundall, and van de Klundert (2014) was to examine how patient portals contribute to health service delivery and patient outcomes. The specific goals were to investigate how outcomes are produced and, most importantly, how variations in outcomes can be explained. The research team started with an exploratory review of background documents and research studies to identify ways in which patient portals may contribute to health service delivery and patient outcomes. The authors identified six main ways which represent “educated guesses” to be tested against the data in the evaluation studies. These studies were identified through a formal and systematic search in four databases between 2003 and 2013. Two members of the research team selected the articles using a pre-established list of inclusion and exclusion criteria and following a two-step procedure. The authors then extracted data from the selected articles and created several tables, one for each outcome category. They organized information to bring forward those mechanisms where patient portals contribute to outcomes and the variation in outcomes across different contexts.

9.3.6. Critical Reviews

Lastly, critical reviews aim to provide a critical evaluation and interpretive analysis of existing literature on a particular topic of interest to reveal strengths, weaknesses, contradictions, controversies, inconsistencies, and/or other important issues with respect to theories, hypotheses, research methods or results ( Baumeister & Leary, 1997 ; Kirkevold, 1997 ). Unlike other review types, critical reviews attempt to take a reflective account of the research that has been done in a particular area of interest, and assess its credibility by using appraisal instruments or critical interpretive methods. In this way, critical reviews attempt to constructively inform other scholars about the weaknesses of prior research and strengthen knowledge development by giving focus and direction to studies for further improvement ( Kirkevold, 1997 ).

Kitsiou, Paré, and Jaana (2013) provide an example of a critical review that assessed the methodological quality of prior systematic reviews of home telemonitoring studies for chronic patients. The authors conducted a comprehensive search on multiple databases to identify eligible reviews and subsequently used a validated instrument to conduct an in-depth quality appraisal. Results indicate that the majority of systematic reviews in this particular area suffer from important methodological flaws and biases that impair their internal validity and limit their usefulness for clinical and decision-making purposes. To this end, they provide a number of recommendations to strengthen knowledge development towards improving the design and execution of future reviews on home telemonitoring.

9.4. Summary

Table 9.1 outlines the main types of literature reviews that were described in the previous sub-sections and summarizes the main characteristics that distinguish one review type from another. It also includes key references to methodological guidelines and useful sources that can be used by eHealth scholars and researchers for planning and developing reviews.

Typology of Literature Reviews (adapted from Paré et al., 2015).

As shown in Table 9.1 , each review type addresses different kinds of research questions or objectives, which subsequently define and dictate the methods and approaches that need to be used to achieve the overarching goal(s) of the review. For example, in the case of narrative reviews, there is greater flexibility in searching and synthesizing articles ( Green et al., 2006 ). Researchers are often relatively free to use a diversity of approaches to search, identify, and select relevant scientific articles, describe their operational characteristics, present how the individual studies fit together, and formulate conclusions. On the other hand, systematic reviews are characterized by their high level of systematicity, rigour, and use of explicit methods, based on an “a priori” review plan that aims to minimize bias in the analysis and synthesis process (Higgins & Green, 2008). Some reviews are exploratory in nature (e.g., scoping/mapping reviews), whereas others may be conducted to discover patterns (e.g., descriptive reviews) or involve a synthesis approach that may include the critical analysis of prior research ( Paré et al., 2015 ). Hence, in order to select the most appropriate type of review, it is critical to know before embarking on a review project, why the research synthesis is conducted and what type of methods are best aligned with the pursued goals.

9.5. Concluding Remarks

In light of the increased use of evidence-based practice and research generating stronger evidence ( Grady et al., 2011 ; Lyden et al., 2013 ), review articles have become essential tools for summarizing, synthesizing, integrating or critically appraising prior knowledge in the eHealth field. As mentioned earlier, when rigorously conducted review articles represent powerful information sources for eHealth scholars and practitioners looking for state-of-the-art evidence. The typology of literature reviews we used herein will allow eHealth researchers, graduate students and practitioners to gain a better understanding of the similarities and differences between review types.

We must stress that this classification scheme does not privilege any specific type of review as being of higher quality than another ( Paré et al., 2015 ). As explained above, each type of review has its own strengths and limitations. Having said that, we realize that the methodological rigour of any review — be it qualitative, quantitative or mixed — is a critical aspect that should be considered seriously by prospective authors. In the present context, the notion of rigour refers to the reliability and validity of the review process described in section 9.2. For one thing, reliability is related to the reproducibility of the review process and steps, which is facilitated by a comprehensive documentation of the literature search process, extraction, coding and analysis performed in the review. Whether the search is comprehensive or not, whether it involves a methodical approach for data extraction and synthesis or not, it is important that the review documents in an explicit and transparent manner the steps and approach that were used in the process of its development. Next, validity characterizes the degree to which the review process was conducted appropriately. It goes beyond documentation and reflects decisions related to the selection of the sources, the search terms used, the period of time covered, the articles selected in the search, and the application of backward and forward searches ( vom Brocke et al., 2009 ). In short, the rigour of any review article is reflected by the explicitness of its methods (i.e., transparency) and the soundness of the approach used. We refer those interested in the concepts of rigour and quality to the work of Templier and Paré (2015) which offers a detailed set of methodological guidelines for conducting and evaluating various types of review articles.

To conclude, our main objective in this chapter was to demystify the various types of literature reviews that are central to the continuous development of the eHealth field. It is our hope that our descriptive account will serve as a valuable source for those conducting, evaluating or using reviews in this important and growing domain.

• Ammenwerth E., de Keizer N. An inventory of evaluation studies of information technology in health care. Trends in evaluation research, 1982-2002. International Journal of Medical Informatics. 2004; 44 (1):44–56. [ PubMed : 15778794 ]
• Anderson S., Allen P., Peckham S., Goodwin N. Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services. Health Research Policy and Systems. 2008; 6 (7):1–12. [ PMC free article : PMC2500008 ] [ PubMed : 18613961 ] [ CrossRef ]
• Archer N., Fevrier-Thomas U., Lokker C., McKibbon K. A., Straus S.E. Personal health records: a scoping review. Journal of American Medical Informatics Association. 2011; 18 (4):515–522. [ PMC free article : PMC3128401 ] [ PubMed : 21672914 ]
• Arksey H., O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005; 8 (1):19–32.
• A systematic, tool-supported method for conducting literature reviews in information systems. Paper presented at the Proceedings of the 19th European Conference on Information Systems ( ecis 2011); June 9 to 11; Helsinki, Finland. 2011.
• Baumeister R. F., Leary M.R. Writing narrative literature reviews. Review of General Psychology. 1997; 1 (3):311–320.
• Becker L. A., Oxman A.D. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Overviews of reviews; pp. 607–631.
• Borenstein M., Hedges L., Higgins J., Rothstein H. Introduction to meta-analysis. Hoboken, nj : John Wiley & Sons Inc; 2009.
• Cook D. J., Mulrow C. D., Haynes B. Systematic reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine. 1997; 126 (5):376–380. [ PubMed : 9054282 ]
• Cooper H., Hedges L.V. In: The handbook of research synthesis and meta-analysis. 2nd ed. Cooper H., Hedges L. V., Valentine J. C., editors. New York: Russell Sage Foundation; 2009. Research synthesis as a scientific process; pp. 3–17.
• Cooper H. M. Organizing knowledge syntheses: A taxonomy of literature reviews. Knowledge in Society. 1988; 1 (1):104–126.
• Cronin P., Ryan F., Coughlan M. Undertaking a literature review: a step-by-step approach. British Journal of Nursing. 2008; 17 (1):38–43. [ PubMed : 18399395 ]
• Darlow S., Wen K.Y. Development testing of mobile health interventions for cancer patient self-management: A review. Health Informatics Journal. 2015 (online before print). [ PubMed : 25916831 ] [ CrossRef ]
• Daudt H. M., van Mossel C., Scott S.J. Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. bmc Medical Research Methodology. 2013; 13 :48. [ PMC free article : PMC3614526 ] [ PubMed : 23522333 ] [ CrossRef ]
• Davies P. The relevance of systematic reviews to educational policy and practice. Oxford Review of Education. 2000; 26 (3-4):365–378.
• Deeks J. J., Higgins J. P. T., Altman D.G. In: Cochrane handbook for systematic reviews of interventions. Higgins J. P. T., Green S., editors. Hoboken, nj : John Wiley & Sons, Ltd; 2008. Analysing data and undertaking meta-analyses; pp. 243–296.
• Deshazo J. P., Lavallie D. L., Wolf F.M. Publication trends in the medical informatics literature: 20 years of “Medical Informatics” in mesh . bmc Medical Informatics and Decision Making. 2009; 9 :7. [ PMC free article : PMC2652453 ] [ PubMed : 19159472 ] [ CrossRef ]
• Dixon-Woods M., Agarwal S., Jones D., Young B., Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. Journal of Health Services Research and Policy. 2005; 10 (1):45–53. [ PubMed : 15667704 ]
• Finfgeld-Connett D., Johnson E.D. Literature search strategies for conducting knowledge-building and theory-generating qualitative systematic reviews. Journal of Advanced Nursing. 2013; 69 (1):194–204. [ PMC free article : PMC3424349 ] [ PubMed : 22591030 ]
• Grady B., Myers K. M., Nelson E. L., Belz N., Bennett L., Carnahan L. … Guidelines Working Group. Evidence-based practice for telemental health. Telemedicine Journal and E Health. 2011; 17 (2):131–148. [ PubMed : 21385026 ]
• Green B. N., Johnson C. D., Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine. 2006; 5 (3):101–117. [ PMC free article : PMC2647067 ] [ PubMed : 19674681 ]
• Greenhalgh T., Wong G., Westhorp G., Pawson R. Protocol–realist and meta-narrative evidence synthesis: evolving standards ( rameses ). bmc Medical Research Methodology. 2011; 11 :115. [ PMC free article : PMC3173389 ] [ PubMed : 21843376 ]
• Gurol-Urganci I., de Jongh T., Vodopivec-Jamsek V., Atun R., Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database System Review. 2013; 12 cd 007458. [ PMC free article : PMC6485985 ] [ PubMed : 24310741 ] [ CrossRef ]
• Hart C. Doing a literature review: Releasing the social science research imagination. London: SAGE Publications; 1998.
• Higgins J. P. T., Green S., editors. Cochrane handbook for systematic reviews of interventions: Cochrane book series. Hoboken, nj : Wiley-Blackwell; 2008.
• Jesson J., Matheson L., Lacey F.M. Doing your literature review: traditional and systematic techniques. Los Angeles & London: SAGE Publications; 2011.
• King W. R., He J. Understanding the role and methods of meta-analysis in IS research. Communications of the Association for Information Systems. 2005; 16 :1.
• Kirkevold M. Integrative nursing research — an important strategy to further the development of nursing science and nursing practice. Journal of Advanced Nursing. 1997; 25 (5):977–984. [ PubMed : 9147203 ]
• Kitchenham B., Charters S. ebse Technical Report Version 2.3. Keele & Durham. uk : Keele University & University of Durham; 2007. Guidelines for performing systematic literature reviews in software engineering.
• Kitsiou S., Paré G., Jaana M. Systematic reviews and meta-analyses of home telemonitoring interventions for patients with chronic diseases: a critical assessment of their methodological quality. Journal of Medical Internet Research. 2013; 15 (7):e150. [ PMC free article : PMC3785977 ] [ PubMed : 23880072 ]
• Kitsiou S., Paré G., Jaana M. Effects of home telemonitoring interventions on patients with chronic heart failure: an overview of systematic reviews. Journal of Medical Internet Research. 2015; 17 (3):e63. [ PMC free article : PMC4376138 ] [ PubMed : 25768664 ]
• Levac D., Colquhoun H., O’Brien K. K. Scoping studies: advancing the methodology. Implementation Science. 2010; 5 (1):69. [ PMC free article : PMC2954944 ] [ PubMed : 20854677 ]
• Levy Y., Ellis T.J. A systems approach to conduct an effective literature review in support of information systems research. Informing Science. 2006; 9 :181–211.
• Liberati A., Altman D. G., Tetzlaff J., Mulrow C., Gøtzsche P. C., Ioannidis J. P. A. et al. Moher D. The prisma statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Annals of Internal Medicine. 2009; 151 (4):W-65. [ PubMed : 19622512 ]
• Lyden J. R., Zickmund S. L., Bhargava T. D., Bryce C. L., Conroy M. B., Fischer G. S. et al. McTigue K. M. Implementing health information technology in a patient-centered manner: Patient experiences with an online evidence-based lifestyle intervention. Journal for Healthcare Quality. 2013; 35 (5):47–57. [ PubMed : 24004039 ]
• Mickan S., Atherton H., Roberts N. W., Heneghan C., Tilson J.K. Use of handheld computers in clinical practice: a systematic review. bmc Medical Informatics and Decision Making. 2014; 14 :56. [ PMC free article : PMC4099138 ] [ PubMed : 24998515 ]
• Moher D. The problem of duplicate systematic reviews. British Medical Journal. 2013; 347 (5040) [ PubMed : 23945367 ] [ CrossRef ]
• Montori V. M., Wilczynski N. L., Morgan D., Haynes R. B., Hedges T. Systematic reviews: a cross-sectional study of location and citation counts. bmc Medicine. 2003; 1 :2. [ PMC free article : PMC281591 ] [ PubMed : 14633274 ]
• Mulrow C. D. The medical review article: state of the science. Annals of Internal Medicine. 1987; 106 (3):485–488. [ PubMed : 3813259 ] [ CrossRef ]
• Evidence-based information systems: A decade later. Proceedings of the European Conference on Information Systems ; 2011. Retrieved from http://aisel ​.aisnet.org/cgi/viewcontent ​.cgi?article ​=1221&context ​=ecis2011 .
• Okoli C., Schabram K. A guide to conducting a systematic literature review of information systems research. ssrn Electronic Journal. 2010
• Otte-Trojel T., de Bont A., Rundall T. G., van de Klundert J. How outcomes are achieved through patient portals: a realist review. Journal of American Medical Informatics Association. 2014; 21 (4):751–757. [ PMC free article : PMC4078283 ] [ PubMed : 24503882 ]
• Paré G., Trudel M.-C., Jaana M., Kitsiou S. Synthesizing information systems knowledge: A typology of literature reviews. Information & Management. 2015; 52 (2):183–199.
• Patsopoulos N. A., Analatos A. A., Ioannidis J.P. A. Relative citation impact of various study designs in the health sciences. Journal of the American Medical Association. 2005; 293 (19):2362–2366. [ PubMed : 15900006 ]
• Paul M. M., Greene C. M., Newton-Dame R., Thorpe L. E., Perlman S. E., McVeigh K. H., Gourevitch M.N. The state of population health surveillance using electronic health records: A narrative review. Population Health Management. 2015; 18 (3):209–216. [ PubMed : 25608033 ]
• Pawson R. Evidence-based policy: a realist perspective. London: SAGE Publications; 2006.
• Pawson R., Greenhalgh T., Harvey G., Walshe K. Realist review—a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005; 10 (Suppl 1):21–34. [ PubMed : 16053581 ]
• Petersen K., Vakkalanka S., Kuzniarz L. Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology. 2015; 64 :1–18.
• Petticrew M., Roberts H. Systematic reviews in the social sciences: A practical guide. Malden, ma : Blackwell Publishing Co; 2006.
• Rousseau D. M., Manning J., Denyer D. Evidence in management and organizational science: Assembling the field’s full weight of scientific knowledge through syntheses. The Academy of Management Annals. 2008; 2 (1):475–515.
• Rowe F. What literature review is not: diversity, boundaries and recommendations. European Journal of Information Systems. 2014; 23 (3):241–255.
• Shea B. J., Hamel C., Wells G. A., Bouter L. M., Kristjansson E., Grimshaw J. et al. Boers M. amstar is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. 2009; 62 (10):1013–1020. [ PubMed : 19230606 ]
• Shepperd S., Lewin S., Straus S., Clarke M., Eccles M. P., Fitzpatrick R. et al. Sheikh A. Can we systematically review studies that evaluate complex interventions? PLoS Medicine. 2009; 6 (8):e1000086. [ PMC free article : PMC2717209 ] [ PubMed : 19668360 ]
• Silva B. M., Rodrigues J. J., de la Torre Díez I., López-Coronado M., Saleem K. Mobile-health: A review of current state in 2015. Journal of Biomedical Informatics. 2015; 56 :265–272. [ PubMed : 26071682 ]
• Smith V., Devane D., Begley C., Clarke M. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. bmc Medical Research Methodology. 2011; 11 (1):15. [ PMC free article : PMC3039637 ] [ PubMed : 21291558 ]
• Sylvester A., Tate M., Johnstone D. Beyond synthesis: re-presenting heterogeneous research literature. Behaviour & Information Technology. 2013; 32 (12):1199–1215.
• Templier M., Paré G. A framework for guiding and evaluating literature reviews. Communications of the Association for Information Systems. 2015; 37 (6):112–137.
• Thomas J., Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. bmc Medical Research Methodology. 2008; 8 (1):45. [ PMC free article : PMC2478656 ] [ PubMed : 18616818 ]
• Reconstructing the giant: on the importance of rigour in documenting the literature search process. Paper presented at the Proceedings of the 17th European Conference on Information Systems ( ecis 2009); Verona, Italy. 2009.
• Webster J., Watson R.T. Analyzing the past to prepare for the future: Writing a literature review. Management Information Systems Quarterly. 2002; 26 (2):11.
• Whitlock E. P., Lin J. S., Chou R., Shekelle P., Robinson K.A. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine. 2008; 148 (10):776–782. [ PubMed : 18490690 ]

This publication is licensed under a Creative Commons License, Attribution-Noncommercial 4.0 International License (CC BY-NC 4.0): see https://creativecommons.org/licenses/by-nc/4.0/

• Cite this Page Paré G, Kitsiou S. Chapter 9 Methods for Literature Reviews. In: Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. Victoria (BC): University of Victoria; 2017 Feb 27.
• PDF version of this title (4.5M)
• Disable Glossary Links

In this Page

• Introduction
• Overview of the Literature Review Process and Steps
• Types of Review Articles and Brief Illustrations
• Concluding Remarks

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Ev... Chapter 9 Methods for Literature Reviews - Handbook of eHealth Evaluation: An Evidence-based Approach

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Encyclopedia of Evidence in Pharmaceutical Public Health and Health Services Research in Pharmacy pp 1–15 Cite as

Methodological Approaches to Literature Review

• Dennis Thomas 2 ,
• Elida Zairina 3 &
• Johnson George 4  
• Living reference work entry
• First Online: 09 May 2023

139 Accesses

The literature review can serve various functions in the contexts of education and research. It aids in identifying knowledge gaps, informing research methodology, and developing a theoretical framework during the planning stages of a research study or project, as well as reporting of review findings in the context of the existing literature. This chapter discusses the methodological approaches to conducting a literature review and offers an overview of different types of reviews. There are various types of reviews, including narrative reviews, scoping reviews, and systematic reviews with reporting strategies such as meta-analysis and meta-synthesis. Review authors should consider the scope of the literature review when selecting a type and method. Being focused is essential for a successful review; however, this must be balanced against the relevance of the review to a broad audience.

• Literature review
• Systematic review
• Meta-analysis
• Scoping review
• Research methodology

This is a preview of subscription content, access via your institution .

Akobeng AK. Principles of evidence based medicine. Arch Dis Child. 2005;90(8):837–40.

CrossRef   CAS   PubMed   PubMed Central   Google Scholar  

Alharbi A, Stevenson M. Refining Boolean queries to identify relevant studies for systematic review updates. J Am Med Inform Assoc. 2020;27(11):1658–66.

CrossRef   PubMed   PubMed Central   Google Scholar  

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

CrossRef   Google Scholar  

Aromataris E MZE. JBI manual for evidence synthesis. 2020.

Google Scholar  

Aromataris E, Pearson A. The systematic review: an overview. Am J Nurs. 2014;114(3):53–8.

CrossRef   PubMed   Google Scholar  

Aromataris E, Riitano D. Constructing a search strategy and searching for evidence. A guide to the literature search for a systematic review. Am J Nurs. 2014;114(5):49–56.

Babineau J. Product review: covidence (systematic review software). J Canad Health Libr Assoc Canada. 2014;35(2):68–71.

Baker JD. The purpose, process, and methods of writing a literature review. AORN J. 2016;103(3):265–9.

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.

Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6(1):1–12.

Brown D. A review of the PubMed PICO tool: using evidence-based practice in health education. Health Promot Pract. 2020;21(4):496–8.

Cargo M, Harris J, Pantoja T, et al. Cochrane qualitative and implementation methods group guidance series – paper 4: methods for assessing evidence on intervention implementation. J Clin Epidemiol. 2018;97:59–69.

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

CrossRef   CAS   PubMed   Google Scholar  

Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med. 1997;127(5):380–7.

Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Cummings SR, Browner WS, Hulley SB, editors. Designing Clinical Research: An Epidemiological Approach. 4th ed. Philadelphia (PA): P Lippincott Williams & Wilkins; 2007. p. 14–22.

Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. JMLA. 2018;106(4):420.

Ferrari R. Writing narrative style literature reviews. Medical Writing. 2015;24(4):230–5.

Flemming K, Booth A, Hannes K, Cargo M, Noyes J. Cochrane qualitative and implementation methods group guidance series – paper 6: reporting guidelines for qualitative, implementation, and process evaluation evidence syntheses. J Clin Epidemiol. 2018;97:79–85.

Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26(2):91–108.

Green BN, Johnson CD, Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med. 2006;5(3):101–17.

Gregory AT, Denniss AR. An introduction to writing narrative and systematic reviews; tasks, tips and traps for aspiring authors. Heart Lung Circ. 2018;27(7):893–8.

Harden A, Thomas J, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. J Clin Epidemiol. 2018;97:70–8.

Harris JL, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis. J Clin Epidemiol. 2018;97:39–48.

Higgins J, Thomas J. In: Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3, updated February 2022). Available from www.training.cochrane.org/handbook.: Cochrane; 2022.

International prospective register of systematic reviews (PROSPERO). Available from https://www.crd.york.ac.uk/prospero/ .

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Landhuis E. Scientific literature: information overload. Nature. 2016;535(7612):457–8.

Lockwood C, Porritt K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, Loveday H, Carrier J, Stannard D. Chapter 2: Systematic reviews of qualitative evidence. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. Available from https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-03 .

Lorenzetti DL, Topfer L-A, Dennett L, Clement F. Value of databases other than medline for rapid health technology assessments. Int J Technol Assess Health Care. 2014;30(2):173–8.

Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group. Preferred reporting items for (SR) and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;6:264–9.

Mulrow CD. Systematic reviews: rationale for systematic reviews. BMJ. 1994;309(6954):597–9.

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143.

Munthe-Kaas HM, Glenton C, Booth A, Noyes J, Lewin S. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol. 2019;19(1):1–13.

Murphy CM. Writing an effective review article. J Med Toxicol. 2012;8(2):89–90.

NHMRC. Guidelines for guidelines: assessing risk of bias. Available at https://nhmrc.gov.au/guidelinesforguidelines/develop/assessing-risk-bias . Last published 29 August 2019. Accessed 29 Aug 2022.

Noyes J, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 1: introduction. J Clin Epidemiol. 2018b;97:35–8.

Noyes J, Booth A, Flemming K, et al. Cochrane qualitative and implementation methods group guidance series – paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. J Clin Epidemiol. 2018a;97:49–58.

Noyes J, Booth A, Moore G, Flemming K, Tunçalp Ö, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods. BMJ Glob Health. 2019;4(Suppl 1):e000893.

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Healthcare. 2015;13(3):141–6.

Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Methods. 2019;10(3):330–42.

CrossRef   PubMed Central   Google Scholar  

Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):1–7.

Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Brit Med J. 2017;358

Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Br Med J. 2016;355

Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008–12.

Tawfik GM, Dila KAS, Mohamed MYF, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health. 2019;47(1):1–9.

The Critical Appraisal Program. Critical appraisal skills program. Available at https://casp-uk.net/ . 2022. Accessed 29 Aug 2022.

The University of Melbourne. Writing a literature review in Research Techniques 2022. Available at https://students.unimelb.edu.au/academic-skills/explore-our-resources/research-techniques/reviewing-the-literature . Accessed 29 Aug 2022.

The Writing Center University of Winconsin-Madison. Learn how to write a literature review in The Writer’s Handbook – Academic Professional Writing. 2022. Available at https://writing.wisc.edu/handbook/assignments/reviewofliterature/ . Accessed 29 Aug 2022.

Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999;18(20):2693–708.

Tricco AC, Lillie E, Zarin W, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16(1):15.

Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Yoneoka D, Henmi M. Clinical heterogeneity in random-effect meta-analysis: between-study boundary estimate problem. Stat Med. 2019;38(21):4131–45.

Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. Am J Gastroenterol. 2009;104(5):1086–92.

Download references

Author information

Authors and affiliations.

Centre of Excellence in Treatable Traits, College of Health, Medicine and Wellbeing, University of Newcastle, Hunter Medical Research Institute Asthma and Breathing Programme, Newcastle, NSW, Australia

Dennis Thomas

Department of Pharmacy Practice, Faculty of Pharmacy, Universitas Airlangga, Surabaya, Indonesia

Elida Zairina

Centre for Medicine Use and Safety, Monash Institute of Pharmaceutical Sciences, Faculty of Pharmacy and Pharmaceutical Sciences, Monash University, Parkville, VIC, Australia

Johnson George

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Johnson George .

Section Editor information

College of Pharmacy, Qatar University, Doha, Qatar

Derek Charles Stewart

Department of Pharmacy, University of Huddersfield, Huddersfield, United Kingdom

Zaheer-Ud-Din Babar

Rights and permissions

Reprints and Permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this entry

Cite this entry.

Thomas, D., Zairina, E., George, J. (2023). Methodological Approaches to Literature Review. In: Encyclopedia of Evidence in Pharmaceutical Public Health and Health Services Research in Pharmacy. Springer, Cham. https://doi.org/10.1007/978-3-030-50247-8_57-1

Download citation

DOI : https://doi.org/10.1007/978-3-030-50247-8_57-1

Received : 22 February 2023

Accepted : 22 February 2023

Published : 09 May 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-50247-8

Online ISBN : 978-3-030-50247-8

eBook Packages : Springer Reference Biomedicine and Life Sciences Reference Module Biomedical and Life Sciences

• Find a journal
• Publish with us

Harvey Cushing/John Hay Whitney Medical Library

• Collections
• Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

• Biomedical Databases
• Global (Public Health) Databases
• Soc. Sci., History, and Law Databases
• Grey Literature
• Trials Registers
• Data and Statistics
• Public Policy
• Google Tips
• Recommended Books
• Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question.  That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment.  Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Discovers relationships between research studies/ideas.
• Identifies major themes, concepts, and researchers on a topic.
• Identifies critical gaps and points of disagreement.
• Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question.  The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

• Make sure your research question is not too broad or too narrow.  Is it manageable?
• Begin writing down terms that are related to your question. These will be useful for searches later.
• If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover? 

• This may depend on your assignment.  How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search. 

Where to find databases:

• use the tabs on this guide
• Find other databases in the Nursing Information Resources web page
• More on the Medical Library web page
• ... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

• Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
• Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
• Review the abstracts of research studies carefully. This will save you time.
• Use the bibliographies and references of research studies you find to locate others.
• Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
• Ask your librarian for help at any time.
• Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

• What was the research question of the study you are reviewing? What were the authors trying to discover?
• Was the research funded by a source that could influence the findings?
• What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
• Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
• If there are conflicting studies, why do you think that is?
• How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips: 

• Review the abstracts carefully.  
• Keep careful notes so that you may track your thought processes during the research process.
• Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
• << Previous: Recommended Books
• Last Updated: Oct 31, 2023 3:00 PM
• URL: https://guides.library.yale.edu/YSNDoctoral

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 5. The Literature Review
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

A literature review surveys prior research published in books, scholarly articles, and any other sources relevant to a particular issue, area of research, or theory, and by so doing, provides a description, summary, and critical evaluation of these works in relation to the research problem being investigated. Literature reviews are designed to provide an overview of sources you have used in researching a particular topic and to demonstrate to your readers how your research fits within existing scholarship about the topic.

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . Fourth edition. Thousand Oaks, CA: SAGE, 2014.

Importance of a Good Literature Review

A literature review may consist of simply a summary of key sources, but in the social sciences, a literature review usually has an organizational pattern and combines both summary and synthesis, often within specific conceptual categories . A summary is a recap of the important information of the source, but a synthesis is a re-organization, or a reshuffling, of that information in a way that informs how you are planning to investigate a research problem. The analytical features of a literature review might:

• Give a new interpretation of old material or combine new with old interpretations,
• Trace the intellectual progression of the field, including major debates,
• Depending on the situation, evaluate the sources and advise the reader on the most pertinent or relevant research, or
• Usually in the conclusion of a literature review, identify where gaps exist in how a problem has been researched to date.

Given this, the purpose of a literature review is to:

• Place each work in the context of its contribution to understanding the research problem being studied.
• Describe the relationship of each work to the others under consideration.
• Identify new ways to interpret prior research.
• Reveal any gaps that exist in the literature.
• Resolve conflicts amongst seemingly contradictory previous studies.
• Identify areas of prior scholarship to prevent duplication of effort.
• Point the way in fulfilling a need for additional research.
• Locate your own research within the context of existing literature [very important].

Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper. 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . Los Angeles, CA: SAGE, 2011; Knopf, Jeffrey W. "Doing a Literature Review." PS: Political Science and Politics 39 (January 2006): 127-132; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012.

Types of Literature Reviews

It is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the primary studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally among scholars that become part of the body of epistemological traditions within the field.

In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews. Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are a number of approaches you could adopt depending upon the type of analysis underpinning your study.

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply embedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to make summary claims of the sort found in systematic reviews [see below].

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses or research problems. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication. This is the most common form of review in the social sciences.

Historical Review Few things rest in isolation from historical precedent. Historical literature reviews focus on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review A review does not always focus on what someone said [findings], but how they came about saying what they say [method of analysis]. Reviewing methods of analysis provides a framework of understanding at different levels [i.e. those of theory, substantive fields, research approaches, and data collection and analysis techniques], how researchers draw upon a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection, and data analysis. This approach helps highlight ethical issues which you should be aware of and consider as you go through your own study.

Systematic Review This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. The goal is to deliberately document, critically evaluate, and summarize scientifically all of the research about a clearly defined research problem . Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?" This type of literature review is primarily applied to examining prior research studies in clinical medicine and allied health fields, but it is increasingly being used in the social sciences.

Theoretical Review The purpose of this form is to examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

NOTE : Most often the literature review will incorporate some combination of types. For example, a review that examines literature supporting or refuting an argument, assumption, or philosophical problem related to the research problem will also need to include writing supported by sources that establish the history of these arguments in the literature.

Baumeister, Roy F. and Mark R. Leary. "Writing Narrative Literature Reviews."  Review of General Psychology 1 (September 1997): 311-320; Mark R. Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Kennedy, Mary M. "Defining a Literature." Educational Researcher 36 (April 2007): 139-147; Petticrew, Mark and Helen Roberts. Systematic Reviews in the Social Sciences: A Practical Guide . Malden, MA: Blackwell Publishers, 2006; Torracro, Richard. "Writing Integrative Literature Reviews: Guidelines and Examples." Human Resource Development Review 4 (September 2005): 356-367; Rocco, Tonette S. and Maria S. Plakhotnik. "Literature Reviews, Conceptual Frameworks, and Theoretical Frameworks: Terms, Functions, and Distinctions." Human Ressource Development Review 8 (March 2008): 120-130; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

Structure and Writing Style

I.  Thinking About Your Literature Review

The structure of a literature review should include the following in support of understanding the research problem :

• An overview of the subject, issue, or theory under consideration, along with the objectives of the literature review,
• Division of works under review into themes or categories [e.g. works that support a particular position, those against, and those offering alternative approaches entirely],
• An explanation of how each work is similar to and how it varies from the others,
• Conclusions as to which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of their area of research.

The critical evaluation of each work should consider :

• Provenance -- what are the author's credentials? Are the author's arguments supported by evidence [e.g. primary historical material, case studies, narratives, statistics, recent scientific findings]?
• Methodology -- were the techniques used to identify, gather, and analyze the data appropriate to addressing the research problem? Was the sample size appropriate? Were the results effectively interpreted and reported?
• Objectivity -- is the author's perspective even-handed or prejudicial? Is contrary data considered or is certain pertinent information ignored to prove the author's point?
• Persuasiveness -- which of the author's theses are most convincing or least convincing?
• Validity -- are the author's arguments and conclusions convincing? Does the work ultimately contribute in any significant way to an understanding of the subject?

II.  Development of the Literature Review

Four Basic Stages of Writing 1.  Problem formulation -- which topic or field is being examined and what are its component issues? 2.  Literature search -- finding materials relevant to the subject being explored. 3.  Data evaluation -- determining which literature makes a significant contribution to the understanding of the topic. 4.  Analysis and interpretation -- discussing the findings and conclusions of pertinent literature.

Consider the following issues before writing the literature review: Clarify If your assignment is not specific about what form your literature review should take, seek clarification from your professor by asking these questions: 1.  Roughly how many sources would be appropriate to include? 2.  What types of sources should I review (books, journal articles, websites; scholarly versus popular sources)? 3.  Should I summarize, synthesize, or critique sources by discussing a common theme or issue? 4.  Should I evaluate the sources in any way beyond evaluating how they relate to understanding the research problem? 5.  Should I provide subheadings and other background information, such as definitions and/or a history? Find Models Use the exercise of reviewing the literature to examine how authors in your discipline or area of interest have composed their literature review sections. Read them to get a sense of the types of themes you might want to look for in your own research or to identify ways to organize your final review. The bibliography or reference section of sources you've already read, such as required readings in the course syllabus, are also excellent entry points into your own research. Narrow the Topic The narrower your topic, the easier it will be to limit the number of sources you need to read in order to obtain a good survey of relevant resources. Your professor will probably not expect you to read everything that's available about the topic, but you'll make the act of reviewing easier if you first limit scope of the research problem. A good strategy is to begin by searching the USC Libraries Catalog for recent books about the topic and review the table of contents for chapters that focuses on specific issues. You can also review the indexes of books to find references to specific issues that can serve as the focus of your research. For example, a book surveying the history of the Israeli-Palestinian conflict may include a chapter on the role Egypt has played in mediating the conflict, or look in the index for the pages where Egypt is mentioned in the text. Consider Whether Your Sources are Current Some disciplines require that you use information that is as current as possible. This is particularly true in disciplines in medicine and the sciences where research conducted becomes obsolete very quickly as new discoveries are made. However, when writing a review in the social sciences, a survey of the history of the literature may be required. In other words, a complete understanding the research problem requires you to deliberately examine how knowledge and perspectives have changed over time. Sort through other current bibliographies or literature reviews in the field to get a sense of what your discipline expects. You can also use this method to explore what is considered by scholars to be a "hot topic" and what is not.

III.  Ways to Organize Your Literature Review

Chronology of Events If your review follows the chronological method, you could write about the materials according to when they were published. This approach should only be followed if a clear path of research building on previous research can be identified and that these trends follow a clear chronological order of development. For example, a literature review that focuses on continuing research about the emergence of German economic power after the fall of the Soviet Union. By Publication Order your sources by publication chronology, then, only if the order demonstrates a more important trend. For instance, you could order a review of literature on environmental studies of brown fields if the progression revealed, for example, a change in the soil collection practices of the researchers who wrote and/or conducted the studies. Thematic [“conceptual categories”] A thematic literature review is the most common approach to summarizing prior research in the social and behavioral sciences. Thematic reviews are organized around a topic or issue, rather than the progression of time, although the progression of time may still be incorporated into a thematic review. For example, a review of the Internet’s impact on American presidential politics could focus on the development of online political satire. While the study focuses on one topic, the Internet’s impact on American presidential politics, it would still be organized chronologically reflecting technological developments in media. The difference in this example between a "chronological" and a "thematic" approach is what is emphasized the most: themes related to the role of the Internet in presidential politics. Note that more authentic thematic reviews tend to break away from chronological order. A review organized in this manner would shift between time periods within each section according to the point being made. Methodological A methodological approach focuses on the methods utilized by the researcher. For the Internet in American presidential politics project, one methodological approach would be to look at cultural differences between the portrayal of American presidents on American, British, and French websites. Or the review might focus on the fundraising impact of the Internet on a particular political party. A methodological scope will influence either the types of documents in the review or the way in which these documents are discussed.

Other Sections of Your Literature Review Once you've decided on the organizational method for your literature review, the sections you need to include in the paper should be easy to figure out because they arise from your organizational strategy. In other words, a chronological review would have subsections for each vital time period; a thematic review would have subtopics based upon factors that relate to the theme or issue. However, sometimes you may need to add additional sections that are necessary for your study, but do not fit in the organizational strategy of the body. What other sections you include in the body is up to you. However, only include what is necessary for the reader to locate your study within the larger scholarship about the research problem.

Here are examples of other sections, usually in the form of a single paragraph, you may need to include depending on the type of review you write:

• Current Situation : Information necessary to understand the current topic or focus of the literature review.
• Sources Used : Describes the methods and resources [e.g., databases] you used to identify the literature you reviewed.
• History : The chronological progression of the field, the research literature, or an idea that is necessary to understand the literature review, if the body of the literature review is not already a chronology.
• Selection Methods : Criteria you used to select (and perhaps exclude) sources in your literature review. For instance, you might explain that your review includes only peer-reviewed [i.e., scholarly] sources.
• Standards : Description of the way in which you present your information.
• Questions for Further Research : What questions about the field has the review sparked? How will you further your research as a result of the review?

IV.  Writing Your Literature Review

Once you've settled on how to organize your literature review, you're ready to write each section. When writing your review, keep in mind these issues.

Use Evidence A literature review section is, in this sense, just like any other academic research paper. Your interpretation of the available sources must be backed up with evidence [citations] that demonstrates that what you are saying is valid. Be Selective Select only the most important points in each source to highlight in the review. The type of information you choose to mention should relate directly to the research problem, whether it is thematic, methodological, or chronological. Related items that provide additional information, but that are not key to understanding the research problem, can be included in a list of further readings . Use Quotes Sparingly Some short quotes are appropriate if you want to emphasize a point, or if what an author stated cannot be easily paraphrased. Sometimes you may need to quote certain terminology that was coined by the author, is not common knowledge, or taken directly from the study. Do not use extensive quotes as a substitute for using your own words in reviewing the literature. Summarize and Synthesize Remember to summarize and synthesize your sources within each thematic paragraph as well as throughout the review. Recapitulate important features of a research study, but then synthesize it by rephrasing the study's significance and relating it to your own work and the work of others. Keep Your Own Voice While the literature review presents others' ideas, your voice [the writer's] should remain front and center. For example, weave references to other sources into what you are writing but maintain your own voice by starting and ending the paragraph with your own ideas and wording. Use Caution When Paraphrasing When paraphrasing a source that is not your own, be sure to represent the author's information or opinions accurately and in your own words. Even when paraphrasing an author’s work, you still must provide a citation to that work.

V.  Common Mistakes to Avoid

These are the most common mistakes made in reviewing social science research literature.

• Sources in your literature review do not clearly relate to the research problem;
• You do not take sufficient time to define and identify the most relevant sources to use in the literature review related to the research problem;
• Relies exclusively on secondary analytical sources rather than including relevant primary research studies or data;
• Uncritically accepts another researcher's findings and interpretations as valid, rather than examining critically all aspects of the research design and analysis;
• Does not describe the search procedures that were used in identifying the literature to review;
• Reports isolated statistical results rather than synthesizing them in chi-squared or meta-analytic methods; and,
• Only includes research that validates assumptions and does not consider contrary findings and alternative interpretations found in the literature.

Cook, Kathleen E. and Elise Murowchick. “Do Literature Review Skills Transfer from One Course to Another?” Psychology Learning and Teaching 13 (March 2014): 3-11; Fink, Arlene. Conducting Research Literature Reviews: From the Internet to Paper . 2nd ed. Thousand Oaks, CA: Sage, 2005; Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1998; Jesson, Jill. Doing Your Literature Review: Traditional and Systematic Techniques . London: SAGE, 2011; Literature Review Handout. Online Writing Center. Liberty University; Literature Reviews. The Writing Center. University of North Carolina; Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: SAGE, 2016; Ridley, Diana. The Literature Review: A Step-by-Step Guide for Students . 2nd ed. Los Angeles, CA: SAGE, 2012; Randolph, Justus J. “A Guide to Writing the Dissertation Literature Review." Practical Assessment, Research, and Evaluation. vol. 14, June 2009; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016; Taylor, Dena. The Literature Review: A Few Tips On Conducting It. University College Writing Centre. University of Toronto; Writing a Literature Review. Academic Skills Centre. University of Canberra.

Writing Tip

Break Out of Your Disciplinary Box!

Thinking interdisciplinarily about a research problem can be a rewarding exercise in applying new ideas, theories, or concepts to an old problem. For example, what might cultural anthropologists say about the continuing conflict in the Middle East? In what ways might geographers view the need for better distribution of social service agencies in large cities than how social workers might study the issue? You don’t want to substitute a thorough review of core research literature in your discipline for studies conducted in other fields of study. However, particularly in the social sciences, thinking about research problems from multiple vectors is a key strategy for finding new solutions to a problem or gaining a new perspective. Consult with a librarian about identifying research databases in other disciplines; almost every field of study has at least one comprehensive database devoted to indexing its research literature.

Frodeman, Robert. The Oxford Handbook of Interdisciplinarity . New York: Oxford University Press, 2010.

Another Writing Tip

Don't Just Review for Content!

While conducting a review of the literature, maximize the time you devote to writing this part of your paper by thinking broadly about what you should be looking for and evaluating. Review not just what scholars are saying, but how are they saying it. Some questions to ask:

• How are they organizing their ideas?
• What methods have they used to study the problem?
• What theories have been used to explain, predict, or understand their research problem?
• What sources have they cited to support their conclusions?
• How have they used non-textual elements [e.g., charts, graphs, figures, etc.] to illustrate key points?

When you begin to write your literature review section, you'll be glad you dug deeper into how the research was designed and constructed because it establishes a means for developing more substantial analysis and interpretation of the research problem.

Hart, Chris. Doing a Literature Review: Releasing the Social Science Research Imagination . Thousand Oaks, CA: Sage Publications, 1 998.

Yet Another Writing Tip

When Do I Know I Can Stop Looking and Move On?

Here are several strategies you can utilize to assess whether you've thoroughly reviewed the literature:

• Look for repeating patterns in the research findings . If the same thing is being said, just by different people, then this likely demonstrates that the research problem has hit a conceptual dead end. At this point consider: Does your study extend current research?  Does it forge a new path? Or, does is merely add more of the same thing being said?
• Look at sources the authors cite to in their work . If you begin to see the same researchers cited again and again, then this is often an indication that no new ideas have been generated to address the research problem.
• Search Google Scholar to identify who has subsequently cited leading scholars already identified in your literature review [see next sub-tab]. This is called citation tracking and there are a number of sources that can help you identify who has cited whom, particularly scholars from outside of your discipline. Here again, if the same authors are being cited again and again, this may indicate no new literature has been written on the topic.

Onwuegbuzie, Anthony J. and Rebecca Frels. Seven Steps to a Comprehensive Literature Review: A Multimodal and Cultural Approach . Los Angeles, CA: Sage, 2016; Sutton, Anthea. Systematic Approaches to a Successful Literature Review . Los Angeles, CA: Sage Publications, 2016.

• << Previous: Theoretical Framework
• Next: Citation Tracking >>
• Last Updated: Oct 10, 2023 1:30 PM
• URL: https://libguides.usc.edu/writingguide

Research Process :: Step by Step

• Introduction
• Select Topic
• Identify Keywords
• Background Information
• Develop Research Questions
• Refine Topic
• Search Strategy
• Popular Databases
• Evaluate Sources
• Types of Periodicals
• Reading Scholarly Articles
• Primary & Secondary Sources
• Organize / Take Notes
• Writing & Grammar Resources
• Annotated Bibliography
• Literature Review
• Citation Styles
• Paraphrasing
• Privacy / Confidentiality
• Research Process
• Selecting Your Topic
• Identifying Keywords
• Gathering Background Info
• Evaluating Sources

Organize the literature review into sections that present themes or identify trends, including relevant theory. You are not trying to list all the material published, but to synthesize and evaluate it according to the guiding concept of your thesis or research question.  

What is a literature review?

A literature review is an account of what has been published on a topic by accredited scholars and researchers. Occasionally you will be asked to write one as a separate assignment, but more often it is part of the introduction to an essay, research report, or thesis. In writing the literature review, your purpose is to convey to your reader what knowledge and ideas have been established on a topic, and what their strengths and weaknesses are. As a piece of writing, the literature review must be defined by a guiding concept (e.g., your research objective, the problem or issue you are discussing, or your argumentative thesis). It is not just a descriptive list of the material available, or a set of summaries

A literature review must do these things:

• be organized around and related directly to the thesis or research question you are developing
• synthesize results into a summary of what is and is not known
• identify areas of controversy in the literature
• formulate questions that need further research

Ask yourself questions like these:

• What is the specific thesis, problem, or research question that my literature review helps to define?
• What type of literature review am I conducting? Am I looking at issues of theory? methodology? policy? quantitative research (e.g. on the effectiveness of a new procedure)? qualitative research (e.g., studies of loneliness among migrant workers)?
• What is the scope of my literature review? What types of publications am I using (e.g., journals, books, government documents, popular media)? What discipline am I working in (e.g., nursing psychology, sociology, medicine)?
• How good was my information seeking? Has my search been wide enough to ensure I've found all the relevant material? Has it been narrow enough to exclude irrelevant material? Is the number of sources I've used appropriate for the length of my paper?
• Have I critically analyzed the literature I use? Do I follow through a set of concepts and questions, comparing items to each other in the ways they deal with them? Instead of just listing and summarizing items, do I assess them, discussing strengths and weaknesses?
• Have I cited and discussed studies contrary to my perspective?
• Will the reader find my literature review relevant, appropriate, and useful?

Ask yourself questions like these about each book or article you include:

• Has the author formulated a problem/issue?
• Is it clearly defined? Is its significance (scope, severity, relevance) clearly established?
• Could the problem have been approached more effectively from another perspective?
• What is the author's research orientation (e.g., interpretive, critical science, combination)?
• What is the author's theoretical framework (e.g., psychological, developmental, feminist)?
• What is the relationship between the theoretical and research perspectives?
• Has the author evaluated the literature relevant to the problem/issue? Does the author include literature taking positions she or he does not agree with?
• In a research study, how good are the basic components of the study design (e.g., population, intervention, outcome)? How accurate and valid are the measurements? Is the analysis of the data accurate and relevant to the research question? Are the conclusions validly based upon the data and analysis?
• In material written for a popular readership, does the author use appeals to emotion, one-sided examples, or rhetorically-charged language and tone? Is there an objective basis to the reasoning, or is the author merely "proving" what he or she already believes?
• How does the author structure the argument? Can you "deconstruct" the flow of the argument to see whether or where it breaks down logically (e.g., in establishing cause-effect relationships)?
• In what ways does this book or article contribute to our understanding of the problem under study, and in what ways is it useful for practice? What are the strengths and limitations?
• How does this book or article relate to the specific thesis or question I am developing?

Text written by Dena Taylor, Health Sciences Writing Centre, University of Toronto

http://www.writing.utoronto.ca/advice/specific-types-of-writing/literature-review

• << Previous: Annotated Bibliography
• Next: Step 5: Cite Sources >>
• Last Updated: Nov 7, 2023 1:00 PM
• URL: https://libguides.uta.edu/researchprocess

University of Texas Arlington Libraries 702 Planetarium Place · Arlington, TX 76019 · 817-272-3000

• Internet Privacy
• Accessibility
• Problems with a guide? Contact Us.

Research Process

• Brainstorming
• Explore Google This link opens in a new window
• Explore Web Resources
• Explore Background Information
• Explore Books
• Explore Scholarly Articles
• Narrowing a Topic
• Primary and Secondary Resources
• Academic, Popular & Trade Publications
• Scholarly and Peer-Reviewed Journals
• Grey Literature
• Clinical Trials
• Evidence Based Treatment
• Scholarly Research
• Database Research Log
• Search Limits
• Keyword Searching
• Boolean Operators
• Phrase Searching
• Truncation & Wildcard Symbols
• Proximity Searching
• Field Codes
• Subject Terms and Database Thesauri
• Reading a Scientific Article
• Website Evaluation
• Article Keywords and Subject Terms
• Cited References
• Citing Articles
• Related Results
• Search Within Publication
• Database Alerts & RSS Feeds
• Personal Database Accounts
• Persistent URLs
• Literature Gap and Future Research
• Web of Knowledge
• Annual Reviews
• Systematic Reviews & Meta-Analyses
• Finding Seminal Works
• Exhausting the Literature
• Finding Dissertations
• Researching Theoretical Frameworks
• Research Methods & Design
• Tests and Measurements
• Organizing Research & Citations This link opens in a new window
• Scholarly Publication
• Learn the Library This link opens in a new window

Resources for a Literature Review or an Annotated Bibliography

Annotated bibliographies and literature reviews are very common forms of writing. The intent of each is to assist you, as the researcher, in gathering resources, identifying trends and problems in the research field, and analyzing those resources to assist your own research. This type of writing is also very helpful to the reader as it identifies key research articles and synthesizes the information to create a coherent picture in which the reader can place your research. Remember that you only want to include pivotal and influential research in this type of writing – this means you will want to focus on scholarly articles that contain primary research. Though literature reviews and annotated bibliographies accomplish a very similar purpose, they are not written in an identical manner.

According to the Purdue OWL website, an annotated bibliography is “a list of sources (books, journals, websites, periodicals, etc.) one has used for researching a topic. … Therefore an annotated bibliography includes a summary and/or evaluation of each of the sources.” An annotated bibliography is compiled of references and summaries in alphabetical order. For more information see the OWL at Purdue Annotated Bibliographies page. For information about how to find examples of annotated bibliographies, see the Library's Annotated Bibliographies FAQ .

A literature review, on the other hand, is “a summary of what the scientific literature says about your specific topic or question.” A literature review generally organizes references by subject matter, theory type, methodology design, etc. A literature review is generally much more exploratory than an annotated bibliography, and must pull together the information that is presented in many disparate sources to form one, cohesive picture of the research field. For more information see the OWL at Purdue Types of APA Papers page.

How do you go about getting the resources you need to write a literature review or an annotated bibliography? Library databases like EBSCOhost and ProQuest are a great place to start because they contain so many resources on so many different topics, but there are some additional databases that you may want to consider using for these types of assignments. These sub-pages identify resources and research techniques for your literature review.

• OWL at Purdue Annotated Bibliographies
• ASC Annotated Bibliographies FAQ
• OWL at Purdue Types of APA Papers

Additional Resources

For additional information about conducting literature reviews, please see the following resources from the NU Library:

• Aveyard, H. (2010). Doing a literature review in health and social care: A practical guide (2nd edition).Berkshire, GBR: Open University Press. Doing A Literature Review In Health And Social Care : A Practical Guide by Helen Aveyard
• Card, N. (2010). Literature review. In N. Salkind (Ed.), Encyclopedia of research design (pp.726-729). Thousand Oaks, CA: SAGE Publications, Inc.
• Chaffee, S., & Lieberma, D. (2001). The challenge of writing the literature review. In A. Alexander, & W. James Potter (Eds.), How to publish your communication research (pp. 23-47).Thousand Oaks, CA: SAGE Publications, Inc.
• Ford, N. (2012). How to do a literature review. In The essential guide to using the web for research (pp. 53-81). London: SAGE Publications Ltd.
• Literature Review. (2004). In Donna M. Mertens, & John A. McLaughlin (Eds.), Research and evaluation methods in special education (pp. 35-50). Thousand Oaks, CA: SAGE Publications, Inc.
• Literature Reviews. (2001). In Bruce A. Thyer (Ed.), The handbook of social work research methods (pp. 400-413). Thousand Oaks, CA: SAGE Publications, Inc.
• Literature Reviews and Bibliographic Searches. (2006). In V. Desai, & R. Potter (Eds.), Doing development research (pp. 209-222). London, England: SAGEPublications, Ltd.
• Race, R. (2008). Literature review. In L. Given (Ed.), The SAGE encyclopedia of qualitative research methods (pp. 488-490). Thousand Oaks, CA: SAGE Publications, Inc.
• Thomas, D. R., & Hodges, I. D. (2010). Doing a literature review. In Designing and managing your research project: Core skills for social and health research (pp. 105-131). London: SAGE Publications Ltd.
• Writing a Literature Review. (2006). In N. Walliman (Ed.), Social research methods (pp. 182-186). London, England: SAGE Publications, Ltd.

Was this resource helpful?

• << Previous: Persistent URLs
• Next: Literature Gap and Future Research >>
• Last Updated: Nov 13, 2023 3:07 PM
• URL: https://resources.nu.edu/researchprocess

Literature Review: Lit Review Sources

• Lit Review Types
• GRADE System
• Do a Lit Review
• Lit Review Sources

Where do I find information for a literature review?

Research is done by...

...by way of...

...communicated through...

...and organized in...

Types of sources for a review...

• Primary source: Usually a report by the original researchers of a study (unfiltered sources)
• Secondary source: Description or summary by somebody other than the original researcher, e.g. a review article (filtered sources)
• Conceptual/theoretical: Papers concerned with description or analysis of theories or concepts associated with the topic
• Anecdotal/opinion/clinical: Views or opinions about the subject that are not research, review or theoretical (case studies or reports from clinical settings)

A Heirarchy of research information:

Source: SUNY Downstate Medical Center. Medical Research Library of Brooklyn. Evidence Based Medicine Course. A Guide to Research Methods: The Evidence Pyramid: http://library.downstate.edu/EBM2/2100.htm

Life Cycle of Publication

Click image to enlarge

Scientific information has a ‘life cycle’ of its own… it is born as an idea, and then matures and becomes more available to the public. First it appears within the so-called ‘invisible college’ of experts in the field, discussed at conferences and symposia or posted as pre-prints for comments and corrections. Then it appears in the published literature (the primary literature), often as a journal article in a peer-reviewed journal.

Researchers can use the indexing and alerting services of the secondary literature to find out what has been published in a field. Depending on how much information is added by the indexer or abstracter, this may take a few months (though electronic publication has sped up this process). Finally, the information may appear in more popular or reference sources, sometimes called the tertiary literature.

The person beginning a literature search may take this process in reverse: using tertiary sources for general background, then going to the secondary literature to survey what has been published, following up by finding the original (primary) sources, and generating their own research Idea.

(Original content by Wade Lee-Smith)

• << Previous: Do a Lit Review
• Next: Readings >>
• Last Updated: Sep 18, 2023 3:26 PM
• URL: https://libguides.utoledo.edu/litreview

Research Methods and Design

• Action Research
• Case Study Design

Literature Review

• Quantitative Research Methods
• Qualitative Research Methods
• Mixed Methods Study
• Indigenous Research and Ethics This link opens in a new window
• Identifying Empirical Research Articles This link opens in a new window
• Research Ethics and Quality
• Data Literacy
• Get Help with Writing Assignments

A literature review is a discussion of the literature (aka. the "research" or "scholarship") surrounding a certain topic. A good literature review doesn't simply summarize the existing material, but provides thoughtful synthesis and analysis. The purpose of a literature review is to orient your own work within an existing body of knowledge. A literature review may be written as a standalone piece or be included in a larger body of work.

You can read more about literature reviews, what they entail, and how to write one, using the resources below. 

SAGE Research Methods Videos

Am i the only one struggling to write a literature review.

Dr. Zina O'Leary explains the misconceptions and struggles students often have with writing a literature review. She also provides step-by-step guidance on writing a persuasive literature review.

An Introduction to Literature Reviews

Dr. Eric Jensen, Professor of Sociology at the University of Warwick, and Dr. Charles Laurie, Director of Research at Verisk Maplecroft, explain how to write a literature review, and why researchers need to do so. Literature reviews can be stand-alone research or part of a larger project. They communicate the state of academic knowledge on a given topic, specifically detailing what is still unknown.

This is the first video in a whole series about literature reviews. You can find the rest of the series in our SAGE database, Research Methods:

Videos covering research methods and statistics

• Literature Review This chapter in SAGE's Encyclopedia of Research Design describes the types of literature reviews and scientific standards for conducting literature reviews.
• UNC Writing Center: Literature Reviews This handout from the Writing Center at UNC will explain what literature reviews are and offer insights into the form and construction of literature reviews in the humanities, social sciences, and sciences.
• Purdue OWL: Writing a Literature Review The overview of literature reviews comes from Purdue's Online Writing Lab. It explains the basic why, what, and how of writing a literature review.

Organizational Tools for Literature Reviews

One of the most daunting aspects of writing a literature review is organizing your research. There are a variety of strategies that you can use to help you in this task. We've highlighted just a few ways writers keep track of all that information! You can use a combination of these tools or come up with your own organizational process. The key is choosing something that works with your own learning style.

Citation Managers

Citation managers are great tools, in general, for organizing research, but can be especially helpful when writing a literature review. You can keep all of your research in one place, take notes, and organize your materials into different folders or categories. Read more about citations managers here:

• Manage Citations & Sources

Concept Mapping

Some writers use concept mapping (sometimes called flow or bubble charts or "mind maps") to help them visualize the ways in which the research they found connects.

There is no right or wrong way to make a concept map. There are a variety of online tools that can help you create a concept map or you can simply put pen to paper. To read more about concept mapping, take a look at the following help guides:

• Using Concept Maps From Williams College's guide, Literature Review: A Self-guided Tutorial

Synthesis Matrix

A synthesis matrix is is a chart you can use to help you organize your research into thematic categories. By organizing your research into a matrix, like the examples below, can help you visualize the ways in which your sources connect. 

• Walden University Writing Center: Literature Review Matrix Find a variety of literature review matrix examples and templates from Walden University.
• Writing A Literature Review and Using a Synthesis Matrix An example synthesis matrix created by NC State University Writing and Speaking Tutorial Service Tutors. If you would like a copy of this synthesis matrix in a different format, like a Word document, please ask a librarian. CC-BY-SA 3.0
• << Previous: Case Study Design
• Next: Quantitative Research Methods >>
• Last Updated: Nov 16, 2023 8:40 AM

CityU Home - CityU Catalog

Literature Review Research

Literature review, types of literature reviews.

• Finding information
• Additional Resources
• Explains the background of research on a topic
• Demonstrates why a topic is significant to a subject area
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches

Argumentative Review This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. 

Integrative Review Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork.

Systematic Review Uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyze data from the studies that are included in the review. Typically it focuses on a very specific empirical question.

Theoretical Review

Examines the theory that has accumulated in regard to an issue, concept, theory, phenomena. Helps to establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems.

* Kennedy, Mary M. "Defining a Literature."  Educational Researcher  36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

• Next: Finding information >>
• Last Updated: Oct 26, 2023 8:34 PM
• URL: https://guides.lib.odu.edu/literaturereview

• Chester Fritz Library
• Library of the Health Sciences
• Thormodsgard Law Library

Literature Reviews

• Get started

Literature Reviews within a Scholarly Work

Literature reviews as a scholarly work.

• Finding Literature Reviews
• Your Literature Search
• Library Books
• How to Videos
• Communicating & Citing Research
• Bibliography

Literature reviews summarize and analyze what has been written on a particular topic and identify gaps or disagreements in the scholarly work on that topic.

Within a scholarly work, the literature review situates the current work within the larger scholarly conversation and emphasizes how that particular scholarly work contributes to the conversation on the topic. The literature review portion may be as brief as a few paragraphs focusing on a narrow topic area.

When writing this type of literature review, it's helpful to start by identifying sources most relevant to your research question. A citation tracking database such as Web of Science can also help you locate seminal articles on a topic and find out who has more recently cited them. See "Your Literature Search" for more details.

A literature review may itself be a scholarly publication and provide an analysis of what has been written on a particular topic without contributing original research. These types of literature reviews can serve to help keep people updated on a field as well as helping scholars choose a research topic to fill gaps in the knowledge on that topic. Common types include:

Systematic Review

Systematic literature reviews follow specific procedures in some ways similar to setting up an experiment to ensure that future scholars can replicate the same steps. They are also helpful for evaluating data published over multiple studies. Thus, these are common in the medical field and may be used by healthcare providers to help guide diagnosis and treatment decisions. Cochrane Reviews are one example of this type of literature review.

Semi-Systematic Review

When a systematic review is not feasible, a semi-systematic review can help synthesize research on a topic or how a topic has been studied in different fields (Snyder 2019). Rather than focusing on quantitative data, this review type identifies themes, theoretical perspectives, and other qualitative information related to the topic. These types of reviews can be particularly helpful for a historical topic overview, for developing a theoretical model, and for creating a research agenda for a field (Snyder 2019). As with systematic reviews, a search strategy must be developed before conducting the review.

Integrative Review

An integrative review is less systematic and can be helpful for developing a theoretical model or to reconceptualize a topic. As Synder (2019) notes, " This type of review often re quires a more creative collection of data, as the purpose is usually not to cover all articles ever published on the topic but rather to combine perspectives and insights from di ff erent fi elds or research traditions" (p. 336).

Source: Snyder, H. (2019). Literature review as a research methodology: An overview and guidelines. Journal of Business Research. 104. 333-339. doi: 10.1016/j.jbusres.2019.07.039

• << Previous: Get started
• Next: Finding Literature Reviews >>
• Last Updated: Oct 27, 2023 12:59 PM
• URL: https://libguides.und.edu/literature-reviews

Types of Literature Review

There are many types of literature review. The choice of a specific type depends on your research approach and design. The following types of literature review are the most popular in business studies:

Narrative literature review , also referred to as traditional literature review, critiques literature and summarizes the body of a literature. Narrative review also draws conclusions about the topic and identifies gaps or inconsistencies in a body of knowledge. You need to have a sufficiently focused research question to conduct a narrative literature review

Systematic literature review requires more rigorous and well-defined approach compared to most other types of literature review. Systematic literature review is comprehensive and details the timeframe within which the literature was selected. Systematic literature review can be divided into two categories: meta-analysis and meta-synthesis.

When you conduct meta-analysis you take findings from several studies on the same subject and analyze these using standardized statistical procedures. In meta-analysis patterns and relationships are detected and conclusions are drawn. Meta-analysis is associated with deductive research approach.

Meta-synthesis, on the other hand, is based on non-statistical techniques. This technique integrates, evaluates and interprets findings of multiple qualitative research studies. Meta-synthesis literature review is conducted usually when following inductive research approach.

Scoping literature review , as implied by its name is used to identify the scope or coverage of a body of literature on a given topic. It has been noted that “scoping reviews are useful for examining emerging evidence when it is still unclear what other, more specific questions can be posed and valuably addressed by a more precise systematic review.” [1] The main difference between systematic and scoping types of literature review is that, systematic literature review is conducted to find answer to more specific research questions, whereas scoping literature review is conducted to explore more general research question.

Argumentative literature review , as the name implies, examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. It should be noted that a potential for bias is a major shortcoming associated with argumentative literature review.

Integrative literature review reviews , critiques, and synthesizes secondary data about research topic in an integrated way such that new frameworks and perspectives on the topic are generated. If your research does not involve primary data collection and data analysis, then using integrative literature review will be your only option.

Theoretical literature review focuses on a pool of theory that has accumulated in regard to an issue, concept, theory, phenomena. Theoretical literature reviews play an instrumental role in establishing what theories already exist, the relationships between them, to what degree existing theories have been investigated, and to develop new hypotheses to be tested.

At the earlier parts of the literature review chapter, you need to specify the type of your literature review your chose and justify your choice. Your choice of a specific type of literature review should be based upon your research area, research problem and research methods.  Also, you can briefly discuss other most popular types of literature review mentioned above, to illustrate your awareness of them.

[1] Munn, A. et. al. (2018) “Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach” BMC Medical Research Methodology

  John Dudovskiy

Research Methods: Literature Reviews

• Annotated Bibliographies
• Literature Reviews
• Scoping Reviews
• Systematic Reviews
• Scholarship of Teaching and Learning
• Persuasive Arguments
• Subject Specific Methodology

A literature review involves researching, reading, analyzing, evaluating, and summarizing scholarly literature (typically journals and articles) about a specific topic. The results of a literature review may be an entire report or article OR may be part of a article, thesis, dissertation, or grant proposal. A literature review helps the author learn about the history and nature of their topic, and identify research gaps and problems.

Steps & Elements

Problem formulation

• Determine your topic and its components by asking a question
• Research: locate literature related to your topic to identify the gap(s) that can be addressed
• Read: read the articles or other sources of information
• Analyze: assess the findings for relevancy
• Evaluating: determine how the article are relevant to your research and what are the key findings
• Synthesis: write about the key findings and how it is relevant to your research

Elements of a Literature Review

• Summarize subject, issue or theory under consideration, along with objectives of the review
• Divide works under review into categories (e.g. those in support of a particular position, those against, those offering alternative theories entirely)
• Explain how each work is similar to and how it varies from the others
• Conclude which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of an area of research

Writing a Literature Review Resources

• How to Write a Literature Review From the Wesleyan University Library
• Write a Literature Review From the University of California Santa Cruz Library. A Brief overview of a literature review, includes a list of stages for writing a lit review.
• Literature Reviews From the University of North Carolina Writing Center. Detailed information about writing a literature review.
• Undertaking a literature review: a step-by-step approach Cronin, P., Ryan, F., & Coughan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), p.38-43

Literature Review Tutorial

• << Previous: Annotated Bibliographies
• Next: Scoping Reviews >>
• Last Updated: Sep 18, 2023 9:39 AM
• URL: https://guides.auraria.edu/researchmethods

1100 Lawrence Street Denver, CO 80204 303-315-7700 Ask Us Directions

Peer Reviewed

What do we study when we study misinformation? A scoping review of experimental research (2016-2022)

Article metrics.

CrossRef Citations

Altmetric Score

PDF Downloads

We reviewed 555 papers published from 2016–2022 that presented misinformation to participants. We identified several trends in the literature—increasing frequency of misinformation studies over time, a wide variety of topics covered, and a significant focus on COVID-19 misinformation since 2020. We also identified several important shortcomings, including overrepresentation of samples from the United States and Europe and excessive emphasis on short-term consequences of brief, text-based misinformation. Most studies examined belief in misinformation as the primary outcome. While many researchers identified behavioural consequences of misinformation exposure as a pressing concern, we observed a lack of research directly investigating behaviour change. 

School of Applied Psychology, University College Cork, Ireland

School of Psychology, University College Dublin, Ireland

Research Question

• What populations, materials, topics, methods, and outcomes are common in published misinformation research from 2016–2022?

Essay Summary

• The goal of this review was to identify the scope of methods and measures used in assessing the impact of real-world misinformation. 
• We screened 8,469 papers published between 2016 and 2022, finding 555 papers with 759 studies where participants were presented with misinformation. 
• The vast majority of studies included samples from the United States or Europe, used brief text-based misinformation (1–2 sentences), measured belief in the misinformation as a primary outcome, and had no delay between misinformation exposure and measurement of the outcome. 
• The findings highlight certain elements of misinformation research that are currently underrepresented in the literature. In particular, we note the need for more diverse samples, measurement of behaviour change in response to misinformation, and assessment of the longer-term consequences of misinformation exposure. 
• Very few papers directly examined effects of misinformation on behaviour (1%) or behavioural intentions (10%), instead measuring proxy outcomes such as belief or attitudes. Nevertheless, many papers draw conclusions regarding the consequences of misinformation for real-world behaviour.
• We advise caution in inferring behavioural consequences unless behaviours (or behavioural intentions) are explicitly measured. 
• We recommend that policymakers reflect on the specific outcomes they hope to influence and consider whether extant evidence indicates that their efforts are likely to be successful.  

Implications

In this article, we report a scoping review of misinformation research from 2016-2022. A scoping review is a useful evidence synthesis approach that is particularly appropriate when the purpose of the review is to identify knowledge gaps or investigate research conduct across a body of literature (Munn et al., 2018). Our review investigates the methods used in misinformation research since interest in so-called “fake news” spiked in the wake of the 2016 U.S. presidential election and the Brexit referendum vote. While previous publications have reflected critically on the current focus and future pathways for the field (Camargo & Simon, 2022), here we address a simple question: what do we study when we study misinformation? We are interested in the methods, outcomes, and samples that are commonly used in misinformation research and what that might tell us about our focus and blind spots.         

Our review covers studies published from January 2016 to July 2022 and includes any studies where misinformation was presented to participants by researchers. The misinformation had to be related to real-world information (i.e., not simple eyewitness misinformation effects), and the researchers had to measure participants’ response to the misinformation as a primary outcome. As expected, we found an increase in misinformation research over time, from just three studies matching our criteria in 2016 to 312 published in 2021. As the number of studies has grown, so too has the range of topics covered. The three studies published in 2016 all assessed political misinformation, but by 2021, just 35% of studies addressed this issue, while the remainder examined other topics, including climate change, vaccines, nutrition, immigration, and more. COVID-19 became a huge focus for misinformation researchers in late 2020, and our review includes over 200 studies that used COVID-related materials. Below we discuss some implications and recommendations for the field based on our findings.

Call for increased diversity & ecological validity

It has been previously noted that the evidence base for understanding misinformation is skewed by pragmatic decisions affecting the topics that researchers choose to study. For example, Altay and colleagues (2023) argue that misinformation researchers typically focus on social media because it is methodologically convenient, and that this can give rise to the false impression that misinformation is a new phenomenon or one solely confined to the internet. Our findings highlight many other methodological conveniences that affect our understanding of misinformation relating to samples, materials, and experimental design. 

Our findings clearly show that certain populations and types of misinformation are well-represented in the literature. In particular, the majority of studies (78%) drew on samples from the United States or Europe. Though the spread of misinformation is widely recognised as a global phenomenon (Lazer, 2018), countries outside the United States and Europe are underrepresented in misinformation research. We recommend more diverse samples for future studies, as well as studies that assess interventions across multiple countries at once (e.g., Porter & Wood, 2021). Before taking action, policymakers should take note of whether claims regarding the spread of misinformation or the effectiveness of particular interventions have been tested in their jurisdictions and consider whether effects are likely to generalise to other contexts.  

There are growing concerns about large-scale disinformation campaigns and how they may threaten democracies (Nagasko, 2020; Tenove, 2020). For example, research has documented elaborate Russian disinformation campaigns reaching individuals via multiple platforms, delivery methods, and media formats (Erlich & Garner, 2023; Wilson & Starbird, 2020). Our review of the misinformation literature suggests that most studies don’t evaluate conditions that are relevant to these disinformation campaigns. Most studies present misinformation in a very brief format, comprising a single presentation of simple text. Moreover, most studies do not include a delay between presentation and measurement of the outcome. This may be due to ethical concerns, which are, of course, of crucial importance when conducting misinformation research (Greene et al., 2022). Nevertheless, this has implications for policymakers, who may draw on research that does not resemble the real-world conditions in which disinformation campaigns are likely to play out. For example, there is evidence to suggest that repeated exposure can increase the potency of misinformation (Fazio et al., 2022; Pennycook et al., 2018), and some studies have found evidence of misinformation effects strengthening over time (Murphy et al., 2021). These variables are typically studied in isolation, and we,therefore, have an incomplete understanding of how they might interact in large-scale campaigns in the real world. This means that policymakers may make assumptions about which messages are likely to influence citizens based on one or two variables—for example, a news story’s source or the political congeniality of its content—without considering the impact of other potentially interacting variables, such as the delay between information exposure and the target action (e.g., voting in an election) or the number of times an individual is likely to have seen the message. In sum, we would recommend a greater focus on ecologically valid methods to assess misinformation that is presented in multiple formats, across multiple platforms, on repeated occasions, and over a longer time interval. We also encourage future research that is responsive to public and policy-maker concerns with regard to misinformation. For example, a misinformation-related topic that is frequently covered in news media is the looming threat of deepfake technology and the dystopian future it may herald (Broinowski, 2022). However, deepfakes were very rarely examined in the studies we reviewed (nine studies in total). 

Our review contributes to a growing debate as to how we should measure the effectiveness of misinformation interventions. Some have argued that measuring discernment (the ability to distinguish true from false information) is key (Guay et al., 2023). For example, in assessing whether an intervention is effective, we should consider the effects of the intervention on belief in fake news (as most studies naturally do) but also consider effects on belief in  true  news—that is, news items that accurately describe true events. This reflects the idea that while believing and sharing misinformation can lead to obvious dangers, not believing or not sharing true information may also be costly. Interventions that encourage skepticism towards media and news sources might cause substantial harm if they undercut trust in real news, particularly as true news is so much more prevalent than fake news (Acerbi et al., 2022). In our review, less than half of the included studies presented participants with both true and false information. Of those that did present true information, just 15% reported a measure of discernment (7% of all included studies), though there was some indication that this outcome measure has been more commonly reported in recent years. We recommend that future studies consider including both true items and a measure of discernment, particularly when assessing susceptibility to fake news or evaluating an intervention. Furthermore, policymakers should consider the possibility of unintended consequences if interventions aiming to reduce belief in misinformation are employed without due consideration of their effects on trust in news more generally.   

Is misinformation likely to change our behaviour?

 Many of our most pressing social concerns related to misinformation centre on the possibility of false information inciting behaviour change—for example, that political misinformation might have a causal effect on how we decide to vote, or that health misinformation might have a causal effect in refusal of vaccination or treatment. In the current review, we found the most common outcome measure was belief in misinformation (78% of studies), followed by attitudes towards the target of the misinformation (18% of studies). While it is, of course, of interest to examine how misinformation can change beliefs and attitudes, decades of research have shown that information provision is often ineffective at meaningfully changing attitudes (Albarracin & Shavitt, 2018) and even where such an intervention is successful, attitude change is not always sufficient to induce behavioural change (Verplanken & Orbell, 2022). 

When assessing whether misinformation can affect behaviour, previous research has reported mixed results. Loomba et al. (2021) found that exposure to COVID vaccine misinformation reduced intentions to get vaccinated, but other studies have reported null or inconsistent effects (Aftab & Murphy, 2022; de Saint Laurent et al., 2022; Greene & Murphy, 2021; Guess et al., 2020). The current review highlights the small number of studies that have examined offline behavioural intentions (10% of papers reviewed) or offline behaviour itself (< 1% of studies) as an outcome of misinformation exposure. Our findings reveal a mismatch between the stated goals and methodology of research, where many papers conceive of misinformation as a substantial problem and may cite behavioural outcomes (such as vaccine refusal) as the driver of this concern, but the studies instead measure belief. We acknowledge that studying real-world effects of misinformation presents some significant challenges, both practical (we cannot follow people into the voting booth or doctor’s office) and ethical (e.g., if experimental presentation of misinformation has the potential to cause real-world harm to participants or society). Moreover, it can be exceptionally difficult to identify causal links between information exposure and complex behaviours such as voting (Aral & Eckles, 2019). Nevertheless, we recommend that where researchers have an interest in behaviour change, they should endeavour to measure that as part of their study. Where a study has only measured beliefs, attitudes, or sharing intentions, we should refrain from drawing conclusions with regard to behaviour. 

From a policy perspective, those who are concerned about misinformation, such as governments and social media companies, ought to clearly specify whether these concerns relate to beliefs or behaviour, or both. Behaviour change is not the only negative outcome that may result from exposure to misinformation—confusion and distrust in news sources are also significant outcomes that many policymakers may wish to address. We recommend that policymakers reflect on the specific outcomes they hope to influence and consider whether extant evidence indicates that their efforts are likely to be successful. For example, if the goal is to reduce belief in or sharing of misinformation, there may be ample evidence to support a particular plan of action. On the other hand, if the goal is to affect a real-world behaviour such as vaccine uptake, our review suggests that the jury is still out. Policymakers may, therefore, be best advised to lend their support to new research aiming to explicitly address the question of behaviour change in response to misinformation. Specifically, we suggest that funding should be made available by national and international funding bodies to directly evaluate the impacts of misinformation in the real world. 

Finding 1: Studies assessing the effects of misinformation on behaviour are rare.

As shown in Table 1, the most commonly recorded outcome by far was belief in the misinformation presented (78% of studies), followed by attitudes (18.31%). Online behavioural intentions, like intention to share (18.05%) or intention to like or comment on a social media post (5.01%), were more commonly measured than offline behavioural intentions (10.94%), like planning to get vaccinated. A tiny proportion of studies (1.58%) measured actual behaviour and how it may change as a result of misinformation exposure. Even then, just one study (0.13%) assessed real-world behaviour—speed of tapping keys in a lab experiment (Bastick, 2021)—all other studies assessed online behaviour such as sharing of news articles or liking social media posts. Thus, no studies in this review assessed the kind of real-world behaviour targeted by misinformation, such as vaccine uptake or voting behaviour.

Finding 2: Studies in this field  overwhelmingly use short, text-based misinformation. 

The most common format for presenting misinformation was text only (62.71% of included studies), followed by text accompanied by an image (32.41%). Use of other formats was rare; video only (1.84%), text and video (1.32%), images only (< 1%), and audio only (< 1%). 

Of the studies that used textual formats (with or without additional accompanying media), the majority (62.72%) presented between one and two sentences of text. An additional 17.50% presented misinformation in a longer paragraph (more than two sentences), 12.92% presented a page or more, and 6.86% did not specify the length of misinformation text presented.

The most frequent framing for the misinformation presented was news stories (44.27%), misinformation presented with no context (33.47%), and Facebook posts (16.47%). Other less frequent misinformation framing included Twitter posts (7.64%), other social media posts (8.04%), other types of webpages (2.11%), fact checks of news stories (1.58%), and government and public communications (0.26%). 

Very few studies presented doctored media to participants; a small number (1.19%) presented deepfake videos and 1.05% presented other forms of doctored media.

Fin ding 3: Most studies assess outcomes instantly. 

Fewer than 7% of studies reported any delay between exposure to the misinformation and the measurement of outcome ( n  = 52). While many did not specify exactly how long the delay was ( n  = 30), most were less than a week; 1–2 minutes ( n  = 4), 5–10 minutes ( n  = 4), 1 day–1 week ( n  = 13), 3 weeks ( n  = 1) and 1–6 months ( n  = 2).

Finding 4: Most participants were from the USA or Europe. 

The majority of participants sampled were from the United States (49.93%), followed by Europe (28.19%) (see Table 2). All other regions each accounted for 6% or less of the total number of participants sampled, such as East Asia (5.53%), Africa (5.27%) and the Middle East (4.74%). Furthermore, 102 studies (13.26%) did not specify from where they sampled participants.

Finding 5: COVID-19 became a major focus of misinformation research. 

Political misinformation was the most commonly studied topic until 2021, when COVID-related misinformation research became the dominant focus of the field (see Table 3 for a full breakdown of the topics included in the selected studies). Overall, experimental misinformation research is on the rise. Our review included one paper from 2016, 12 papers from 2017, 18 papers from 2018, 48 papers from 2019, 123 papers from 2020, 231 papers from 2021, and 122 papers for the first half of 2022. 

Finding 6: Most studies do not report discernment between true and false misinformation.

In total, 340 studies (45.12%) presented participants with both true and false information. Of these studies, 52 (15.29%) reported a measure of discernment based on participants’ ability to discriminate between true and false information (e.g., the difference in standardised sharing intention scores between true and false items). Across the entire review then, fewer than 7% of studies report discernment between true and false information as an outcome. There was some indication that measurement of discernment is becoming more common over time; no studies included in the review reported a measure of discernment prior to 2019, and 48 out of the 52 studies that did measure discernment were published between 2020 and 2022.

A search was conducted to identify studies that presented participants with misinformation and measured their responses (e.g., belief in misinformation) after participants were exposed to misinformation. All studies must have been published since January 2016, with an English-language version available in a peer-reviewed journal. The final search for relevant records was carried out on the 16th of July, 2022. Searches were carried out in three databases (Scopus, Web of Science, and PsychINFO) using the search terms “misinformation” OR “fake news” OR “disinformation” OR “fabricated news” OR “false news.” The search strategy, inclusion criteria, and extraction templates were preregistered at  https://osf.io/d5hrj/ . 

Inclusion criteria

There were two primary criteria for inclusion in the current scoping review. A study was eligible for inclusion if it (i) presented participants with misinformation with any potential for real-world consequences and (ii) measured participants’ responses to this misinformation (e.g., belief in the misinformation, intentions to share the misinformation) as a main outcome. 

Exclusion criteria

Studies were excluded if they presented participants with misinformation of no real-world consequence (e.g., misinformation about a simulated crime, fabricated stories about fictitious plane crashes, misinformation about fictional persons that were introduced during the course of an experiment). If the misinformation was only relevant within the narrow confines of an experiment, we considered the paper ineligible. Furthermore, studies were excluded if they presented participants with general knowledge statements (e.g., trivia statements) or if they presented participants with misleading claims that were not clearly inaccurate (e.g., a general exaggeration of the benefits of a treatment). Studies were also excluded if the misinformation was only presented in the context of a debunking message, as were studies where the misinformation was presented as a hypothetical statement (e.g., “imagine if we told you that …”, “how many people do you think believe that…”). Studies of eyewitness memory were excluded, as were any studies not published in English. Finally, opinion pieces, commentaries, systematic reviews, or observational studies were excluded.

Originally, only experimental studies were to be included in the review. However, upon screening the studies, it became apparent that distinguishing between experiments, surveys, and intervention-based research was sometimes difficult—for example, cross-sectional studies exploring individual differences in fake news susceptibility might not be classified as true experiments (as they lack control groups and measure outcomes at only one time point), but they were clearly relevant to our aims. To avoid arbitrary decisions, we decided to drop this requirement and instead included all articles that met the inclusion criteria.

Screening and selection process

The search strategy yielded a total of 18,333 records (see Figure 2 for a summary of the screening process). Curious readers may note that a Google Scholar search for the search terms listed above produces a substantially different number of hits, though the number will vary from search to search. This lack of reproducibility in Google Scholar searches is one of many reasons why Google Scholar is not recommended for use in systematic reviews, and the three databases employed here are preferred (Gusenbauer & Haddaway, 2020; also see Boeker et al., 2013; Bramer et al., 2016). Following the removal of duplicates ( n  = 9,864), a total of 8,469 records were eligible to be screened. The titles and abstracts of the 8,469 eligible records were screened by six reviewers, in pairs of two, with a seventh reviewer resolving conflicts where they arose (weighted Cohen’s κ = 0.81). A total of 7,666 records were removed at this stage, as the records did not meet the criteria of the scoping review. 

The full texts of the remaining 803 records were then screened by four reviewers in pairs of two, with conflicts resolved by discussion among the pair with the conflict (weighted Cohen’s κ = 0.68). Among the 803 records, 248 records were excluded (see Figure 2 for reasons for exclusion). Thus, there were 555 papers included for extraction, with a total of 759 studies included therein. An alphabetical list of all included articles is provided in the Appendix, and the full data file listing all included studies and their labels is available at  https://osf.io/3apkt/ .

• / Psychology

Cite this Essay

Murphy, G., de Saint Laurent, C., Reynolds, M., Aftab, O., Hegarty, K. Sun, Y. & Greene, C. M. (2023). What do we study when we study misinformation? A scoping review of experimental research (2016-2022). Harvard Kennedy School (HKS) Misinformation Review . ttps://doi.org/10.37016/mr-2020-130

Bibliography

Acerbi, A., Altay, S., & Mercier, H. (2022). Fighting misinformation or fighting for information?. Harvard Kennedy School (HKS) Misinformation Review, 3 (1). https://doi.org/10.37016/mr-2020-87

Aftab, O., & Murphy, G. (2022). A single exposure to cancer misinformation may not significantly affect related behavioural intentions. HRB Open Research, 5 (82), 82. https://doi.org/10.12688/hrbopenres.13640.1

Albarracin, D., & Shavitt, S. (2018). Attitudes and attitude change. Annual Review of Psychology, 69 , 299–327. https://psycnet.apa.org/doi/10.1146/annurev-psych-122216-011911

Altay, S., Berriche, M., & Acerbi, A. (2023). Misinformation on misinformation: Conceptual and methodological challenges.  Social Media+ Society ,  9 (1), 20563051221150412. https://doi.org/10.1177/20563051221150412

Aral, S., & Eckles, D. (2019). Protecting elections from social media manipulation. Science , 365 (6456), 858–861. https://doi.org/doi:10.1126/science.aaw8243

Bastick, Z. (2021). Would you notice if fake news changed your behavior? An experiment on the unconscious effects of disinformation.  Computers in Human Behavior ,  116 , 106633. https://doi.org/10.1016/j.chb.2020.106633

Bramer, W. M., Giustini, D., & Kramer, B. M. R. (2016). Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: A prospective study. Systematic Reviews , 5 (1), 39. https://doi.org/10.1186/s13643-016-0215-7

Boeker, M., Vach, W., & Motschall, E. (2013). Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC Medical Research Methodology , 13 (1), 131. https://doi.org/10.1186/1471-2288-13-131

Broinowski, A. (2022). Deepfake nightmares, synthetic dreams: A review of dystopian and utopian discourses around deepfakes, and why the collapse of reality may not be imminent—yet.  Journal of Asia-Pacific Pop Culture ,  7 (1), 109–139. https://doi.org/10.5325/jasiapacipopcult.7.1.0109

Camargo, C. Q., & Simon, F. M. (2022). Mis- and disinformation studies are too big to fail: Six suggestions for the field’s future. Harvard Kennedy School (HKS) Misinformation Review , 3 (5). https://doi.org/10.37016/mr-2020-106

de Saint Laurent, C., Murphy, G., Hegarty, K., & Greene, C. M. (2022). Measuring the effects of misinformation exposure and beliefs on behavioural intentions: A COVID-19 vaccination study. Cognitive Research: Principles and Implications , 7 (1), 87. https://doi.org/10.1186/s41235-022-00437-y

Erlich, A., & Garner, C. (2023). Is pro-Kremlin disinformation effective? Evidence from Ukraine. The International Journal of Press/Politics , 28( 1), 5–28. https://doi.org/10.1177/19401612211045221

Fazio, L. K., Pillai, R. M., & Patel, D. (2022). The effects of repetition on belief in naturalistic settings.  Journal of Experimental Psychology: General, 151 (10), 2604–2613. https://doi.org/10.1037/xge0001211

Greene, C. M., de Saint Laurent, C., Murphy, G., Prike, T., Hegarty, K., & Ecker, U. K. (2022). Best practices for ethical conduct of misinformation research: A scoping review and critical commentary. European Psychologist, 28 (3), 139–150. https://doi.org/10.1027/1016-9040/a000491

Greene, C. M., & Murphy, G. (2021). Quantifying the effects of fake news on behavior: Evidence from a study of COVID-19 misinformation. Journal of Experimental Psychology: Applied , 27 (4), 773–784. https://doi.org/10.1037/xap0000371

Guay, B., Berinsky, A. J., Pennycook, G., & Rand, D. (2023). How to think about whether misinformation interventions work. Nature Human Behaviour, 7 , 1231–1233. https://doi.org/10.1038/s41562-023-01667-w

Guess, A. M., Lockett, D., Lyons, B., Montgomery, J. M., Nyhan, B., & Reifler, J. (2020). “Fake news” may have limited effects beyond increasing beliefs in false claims. Harvard Kennedy School (HKS) Misinformation Review , 1 (1). https://doi.org/10.37016/mr-2020-004

Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods , 11 (2), 181–217. https://doi.org/https://doi.org/10.1002/jrsm.1378

Loomba, S., de Figueiredo, A., Piatek, S. J., de Graaf, K., & Larson, H. J. (2021). Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA. Nature Human Behaviour , 5 (3), 337–348. https://doi.org/10.1038/s41562-021-01056-1

Munn, Z., Peters, M. D., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology , 18 . https://doi.org/10.1186/s12874-018-0611-x

Murphy, G., Lynch, L., Loftus, E., & Egan, R. (2021). Push polls increase false memories for fake new stories.  Memory ,  29 (6), 693–707. https://doi.org/10.1080/09658211.2021.1934033

Nagasako, T. (2020). Global disinformation campaigns and legal challenges. International Cybersecurity Law Review, 1 (1–2), 125–136. https://doi.org/10.1365/s43439-020-00010-7

Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., Schudson, M., Sloman, S. A., Sunstein, C. R., Thorson, E. A., Watts, D. J., & Zittrain, J. L. (2018). The science of fake news. Science , 359 (6380), 1094-1096. https://doi.org/doi:10.1126/science.aao2998

Pennycook, G., Cannon, T. D., & Rand, D. G. (2018). Prior exposure increases perceived accuracy of fake news.  Journal of Experimental Psychology: General ,  147 (12), 1865–1880. https://doi.org/10.1037/xge0000465

Porter, E., & Wood, T. J. (2021). The global effectiveness of fact-checking: Evidence from simultaneous experiments in Argentina, Nigeria, South Africa, and the United Kingdom. Proceedings of the National Academy of Sciences , 118 (37), e2104235118. https://doi.org/10.1073/pnas.2104235118

Tenove, C. (2020). Protecting democracy from disinformation: Normative threats and policy responses. The International Journal of Press/Politics , 25 (3), 517–537. https://doi.org/10.1177/1940161220918740

Verplanken, B., & Orbell, S. (2022). Attitudes, habits, and behavior change. Annual Review of Psychology , 73 , 327–352. https://doi.org/10.1146/annurev-psych-020821-011744

Wilson, T., & Starbird, K. (2020). Cross-platform disinformation campaigns: lessons learned and next steps. Harvard Kennedy School (HKS) Misinformation Review , 1 (1) . https://doi.org/10.37016/mr-2020-002

This project was funded by the Health Research Board of Ireland – COV19-2020-030. The funding body had no role in the design, interpretation, or reporting of the research.

Competing Interests

The authors declare no competing interests.

This review protocol was exempt from ethics approval.

This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided that the original author and source are properly credited.

Data Availability

All materials needed to replicate this study are available via the Harvard Dataverse at  https://doi.org/10.7910/DVN/X1YH6S and OSF at https://osf.io/3apkt/ .

• Open access
• Published: 08 November 2023

Childhood vaccine refusal and what to do about it: a systematic review of the ethical literature

• Kerrie Wiley 1 ,
• Maria Christou-Ergos 1 ,
• Chris Degeling 2 ,
• Rosalind McDougall 3 ,
• Penelope Robinson 1 ,
• Katie Attwell 4 ,
• Catherine Helps 1 ,
• Shevaun Drislane 4 &
• Stacy M Carter 2  

BMC Medical Ethics volume  24 , Article number:  96 ( 2023 ) Cite this article

380 Accesses

9 Altmetric

Metrics details

Parental refusal of routine childhood vaccination remains an ethically contested area. This systematic review sought to explore and characterise the normative arguments made about parental refusal of routine vaccination, with the aim of providing researchers, practitioners, and policymakers with a synthesis of current normative literature.

Nine databases covering health and ethics research were searched, and 121 publications identified for the period Jan 1998 to Mar 2022. For articles, source journals were categorised according to Australian Standard Field of Research codes, and normative content was analysed using a framework analytical approach.

Most of the articles were published in biomedical journals (34%), bioethics journals (21%), and journals that carry both classifications (20%). Two central questions dominated the literature: (1) Whether vaccine refusal is justifiable (which we labelled ‘refusal arguments’); and (2) Whether strategies for dealing with those who reject vaccines are justifiable (‘response arguments’). Refusal arguments relied on principlism, religious frameworks, the rights and obligations of parents, the rights of children, the medico-legal best interests of the child standard, and the potential to cause harm to others. Response arguments were broadly divided into arguments about policy, arguments about how individual physicians should practice regarding vaccine rejectors, and both legal precedents and ethical arguments for vaccinating children against a parent’s will. Policy arguments considered the normative significance of coercion, non-medical or conscientious objections, and possible reciprocal social efforts to offset vaccine refusal. Individual physician practice arguments covered nudging and coercive practices, patient dismissal, and the ethical and professional obligations of physicians. Most of the legal precedents discussed were from the American setting, with some from the United Kingdom.

Conclusions

This review provides a comprehensive picture of the scope and substance of normative arguments about vaccine refusal and responses to vaccine refusal. It can serve as a platform for future research to extend the current normative literature, better understand the role of cultural context in normative judgements about vaccination, and more comprehensively translate the nuance of ethical arguments into practice and policy.

Peer Review reports

Introduction

Vaccine rejection has existed for as long as vaccines [ 1 ]. Despite the significant contribution of childhood vaccination to reductions in global child morbidity and mortality [ 2 ], some parents continue to reject vaccines for their children. Parents’ reasons for rejection vary widely, and often depend on their social settings. For example, in high-income settings where around 2–3% of parents reject routine childhood vaccines [ 3 , 4 ], reasons can include previous bad experiences with vaccines or the medical system, concerns about vaccine safety, doubt about the effectiveness or necessity of vaccines, alternative health approaches, and participation in particular social groups or communities. These reasons can be grounded in deeply held religious beliefs or general philosophical approaches to health, views on freedom of choice, or mistrust in government and/or the vested interests of vaccine producers, among other things [ 5 , 6 , 7 , 8 ].

Vaccination plays a dual role in disease prevention: it serves to protect the vaccinated individual from disease, and when vaccination rates reach a high enough threshold for some diseases, also protects the broader community—including those who remain unvaccinated—by disrupting disease transmission through herd immunity. This dual role of vaccination, providing benefit to both the individual and community, complicates ethical questions regarding vaccine refusal, specifically, whether vaccine rejection is ethically justifiable.

Health care providers, communities, and governments encourage uptake and discourage vaccine rejection by various means, and the dual role of vaccination is also relevant to an evaluation of these practice and policy responses. Vaccine acceptance is encouraged with interventions like incentives, health provider recommendations and “nudges” directed at individual families, as well as by facilitating easier access to vaccination through strategies such as cost reduction and making clinic locations and opening times convenient, with many of these interventions supported by varying levels of evidence [ 9 ]. Governments often discourage vaccine rejection via the imposition of mandates that can vary in type and severity [ 10 ] and are not always well-supported by evidence [ 11 ]. These can include punitive measures, such as limiting unvaccinated children’s access to early childhood education or daycare. A thorough understanding of the ethical dimensions of childhood vaccine rejection and responses to it is important when navigating vaccine rejection in the clinical setting, and when formulating policy [ 12 ]. Systematic reviews of the evidence are considered best practice for informing vaccine practice and policy however, to our knowledge there have not yet been any published systematic reviews of the literature on the ethics of childhood vaccine rejection despite there being a broad literature on the subject. We sought to systematically explore and characterise the normative arguments made about parental refusal of routine vaccination, with the aim of better informing vaccine policy and practice.

We searched nine databases for literature that discussed normative positions on childhood vaccine rejection. Refer to the PRISMA flow chart (Fig.  1 .)

PRIMSA Flow Diagram of Review

Search strategy

We searched Medline, Embase, Philosophers Index, Philpapers, Project Muse, Cinahl, The Global Digital Library on Ethics (globethics.net), The Bioethics Literature Database (BELIT), and Pubmed using the general search strategy listed in Fig.  2 for articles published between January 1998 and March 2022.

Inclusion criteria

We included any publication which provided a substantive normative argument about parental refusal of routine vaccines for children aged five and under. We used a broad definition of ‘normative’ to mark anything that goes beyond mere description to consider right and wrong, good and bad, justifiable and unjustifiable, or legitimate and illegitimate actions or ways of being in the world. Our broad conception included textual forms such as ethical reflections, prudential and legal norms, and accounts of rationality. We used ’substantive’ to mark publications where the authors’ main purpose was to make an argument about whether vaccine refusal is morally justifiable. This included empirical research that explicitly examined normative dimensions of vaccine refusal. We were limited to reviewing publications published in English.

Exclusion criteria

We excluded publications where authors made a normative claim in passing, but the publication’s main purpose was to report non-normative empirical findings. We also excluded: publications on adult vaccination (including COVID vaccination) and the HPV vaccine (which is administered in adolescence, not childhood); empirical research such as surveys or interviews, unless they expressly explored normative arguments; and descriptive publications about the characteristics of the anti-vaccination movement that provided no normative position.

Screening and data extraction

After search execution and duplicate removal, a screening triangulation exercise was undertaken to ensure consistency among the screeners. A set of 20 titles and abstracts were independently screened by six authors, and the results compared. The inclusion and exclusion criteria were refined in a subsequent group discussion, and a sub-set of full text articles were then screened and evaluated by the same group of people, and results again compared. A discussion of this second triangulation step resulted in a refined and standardized screening approach.

The authorship group were then divided into four pairs, and the remaining titles and abstracts divided among the pairs. Each individual screened titles and abstracts against inclusion criteria, and then met with their screening partner to compare results and discuss and resolve any differences.

Full text was sought for each record screened for inclusion, and a second screening then removed articles which didn’t meet the inclusion criteria once the full text was read, articles that could not be sourced, and duplicates not identified in the initial screening.

The final list of full text publications was then divided among four authors (SC, RM, CD and KW) for data extraction using the concept of “information units” described by Mertz and colleagues [ 13 ]. In this context an information unit was defined as a normative issue or argument, and each of the four ‘extracting’ authors summarized each of the relevant information units in the papers they were assigned.

For included journal articles, Australian Standard Field of Research (FoR) codes for the journal that each article appeared in were sourced as a proxy for the disciplinary location of the article (e.g. bioethics, medicine, law). We used the Australian and New Zealand Standard Research Classification (ANZSRC) 2008, as this was the current standard when analysis commenced [ 14 ]. We used two digit FoR codes (division codes) to identify the source journal as either being Medical and Health Sciences (code 11), Ethics and Philosophy (code 22) Law (code 18) or other codes grouped as “other”. In some cases, the journal was assigned a combination of these codes (refer to Fig.  3 ).

Respective percentages of included articles falling under various ANZSRC FoR Codes (2008)

Quality assessment in systematic reviews of normative literature remains a contested area, with various options and no established best practice approach [ 15 ]. In this review, we took a satisficing approach to quality appraisal [ 16 ]: publication in a peer-reviewed journal or by a reputable academic publisher was taken as a sufficient level of quality to justify inclusion in the review. The peer review process undergone by PhD theses was also taken to be a sufficient indictor of quality to justify inclusion. Further quality appraisal of individual publications was not undertaken. This aligns with the purpose of the review which was to map and synthesize the current literature on this topic.

A framework approach was used to organise and synthesise the data [ 17 ]. The extracted information units were read by one author (KW), and a coding frame inductively developed to summarise and classify the information units extracted by the group. The publications were then independently coded according to this framework by two authors (KW and PR). Following this, the two authors met and compared their coding, discussing any differences and resolving them by consensus. The data were then synthesized into themes. In addition, for journal articles, the ANZSRC Field of Research codes for the journal each article appeared in were descriptively analysed to assess the distribution of the included literature across various disciplines.

Search results

Five thousand, two hundred and thirty-one publications were returned by the searches (see Fig.  1 ). Eight hundred and twenty-two duplicates were removed in the first instance, leaving 4409 records to be screened by title and abstract. During this screening process 4058 records were excluded, leaving 351 full text publications to be assessed. Of these a further 230 records were excluded (due to not meeting the inclusion criteria, previously unidentified duplicates, or inability to source the full text), leaving 121 publications for inclusion in the review. These included 117 journal articles, three theses and one book.

Literature source type

Analysis of the ANZSRC Field of Research codes of the source journals of included articles revealed three main areas, or a combination of them (Fig.  3 ). Around half were coded to medicine (63%); of these, just over half were dual coded to ethics (20%) or another code (9%). 21% of articles were from the philosophy or ethics literature alone; another 25% were from ethics and medicine or ethics and law. Law was the least dominant discipline, with only 12% of articles being coded to law (alone or in combination with other disciplines). This pattern suggests active concern within medicine regarding non-vaccination, but also widespread overlap in concern between medicine, ethics, and law.

Main themes found in the literature

Articles addressed two central questions (see Table  1 ):

Whether vaccine refusal was justified (henceforth ‘refusal’ arguments).

Whether various policy or practice responses to those who reject vaccines are justified (henceforth ‘response’ arguments).

Descriptive analysis of content

The literature was dominated by papers focused on ‘response’ arguments (61%). A smaller group of papers address ‘refusal’ arguments (19%), and about 18% considered both ‘refusal’ and ‘response’, usually making normative arguments about vaccine refusal as background to arguments regarding ‘response’ (See Fig.  4 ). Less than 2% of papers had a different focus.

Comparative frequencies of themes occurring among included articles

‘Response’ arguments were more common in the medical and health sciences literature (ERA FoR code 11, see Fig.  5 ). Although the ethics/philosophy (FoR code 22) and law literatures (FoR code 18) were also dominated by ‘response’ arguments, these journals—unlike medical journals—were more likely to include ‘refusal’ arguments.

Comparative frequency of overarching themes across the different disciplines of the included articles

As would be expected, authors made ‘response’ and ‘refusal’ arguments in different ways. In the following sections we consider the detail of how arguments were made. We refer to each included article by its unique reference listed in Table  1 .

‘Refusal’ arguments: whether or not vaccine rejection by individual parents is justifiable

Arguments about whether vaccine refusal by individual parents is justifiable included consideration of parents’ rights, the interests of the child (including the legal ‘best interests of the child standard’), the value of herd immunity, the epistemic basis for ethical claims, and the relevance of religious views. Our sampling period included a special issue of Narrative Inquiry in Bioethics which published narratives written by parents to communicate their normative positions on vaccination. Most of these were written by non-vaccinating parents, and they make up over one third of all arguments in the identified literature that support refusal. On balance, most of the literature argues that it is not justifiable for parents to refuse routine vaccination for their children.

Some arguments within the literature were absolute in their position on whether vaccine rejection is justifiable; others weighed competing values in a situation-specific approach. Irrespective of the arguments used to justify a position, most of the literature frames the question of whether vaccine rejection is justifiable based on three key areas of concern: (i) Respect for autonomy, the doctrine of informed consent and the value of liberty, (ii) Consequences for the child and others, and/or (iii) The normative significance of parental trust, distrust, and uncertainty. We explore the main arguments within these concepts below. As the discussion shows, these concepts are not discrete – they are often weighed against one another, linked by causal claims, or held in tension in the arguments made. Figure  6 represents proportionally the ’refusal’ arguments made in the reviewed literature.

‘Refusal’ arguments made in the literature on the ethics of vaccine refusal

Respect for autonomy, the doctrine of informed consent and the value of liberty

Fifteen papers from this sample present arguments that vaccine refusal is justified based on respect for parental autonomy, rights, or liberties (21, 23, 25, 31, 32, 35, 36, 39, 68, 71, 75, 80, 94, 100, 121). Some argue that vaccine refusal is justified on the basis of preserving legal rights (31, 80) or expression of religious freedom [ 23 ]. Opposing positions (including from four of the authors who also offer arguments justifying refusal) argue that, on balance, considerations regarding respect for autonomy are, or can be, outweighed by the potential harm caused to the child and others by not vaccinating though the increased risk of vaccine preventable diseases (21, 36, 20, 23, 110). This includes legal perspectives arguing that the freedom to choose is not unfettered [ 25 ] and that courts can override parental autonomy if this is in the child’s best interest (75, 85), as well as arguments from religious perspectives that the freedom to exercise religious beliefs needs to be weighed against harm caused to others (21,91). Those who argue that vaccine refusal is justified counter that disrespecting parental autonomy can also cause harm to the child through loss of trust and possible disengagement of the child from the healthcare system (100), and that the increased risk of disease is a price worth paying to ensure that political values are preserved (71). Of note: non-vaccinating parents also assert a right to make choices for their children in support of their refusal [ 14 , 18 ], but unlike others, their arguments are based primarily on epistemic claims about vaccine effectiveness, necessity and safety rather than moral or ethical positions. However, they assert that these doubts necessitate respect for their decision.

Consequences for others and the child

Most of the literature argues for or against the justifiability of vaccine refusal based on consequences. These include potential harms from vaccine preventable diseases or vaccines themselves, or conversely, potential benefits from herd immunity. The concept of herd immunity is deployed in different ways. Those justifying vaccine refusal in certain circumstances argue that in settings where there is a high level of herd immunity, the risk posed by an unvaccinated child is not great enough to override respect for parental autonomy (62, 65, 94, 98), and that the benefits of community protection do not justify the individual risk posed by the vaccine and borne by the child who is already protected through herd immunity (72, 96, 97, 17, 93, 108). Perspectives of non-vaccinators echo these ideas by asserting that some diseases are not harmful enough to proscribe vaccine refusal [ 14 ] and that vaccine injury contributes to and justifies refusal [ 16 ].

In contrast, those who argue that refusal is not justifiable propose a duty to contribute to herd immunity because it is a public good (7,80, 19,120, 33, 48, 68,115), or that free-riding (allowing one’s child to enjoy the benefits of herd immunity provided by others, while avoiding the risk of vaccinating) is unfair (37,46, 48). On this account, the vaccine refusal of a few may undermine herd immunity and thus cause harm to the many by increasing disease risks (9, 11, 26, 37, 59, 76, 81, 86); further, these risks are borne by the most vulnerable (43). These arguments about harm to others include those made by authors writing from religious perspectives (8, 81, 84, 92, 98). Finally, an account by a vaccinating parent suggests that harms resulting from non-vaccination are blameworthy because they are an intentional act of aggression against vaccinated children [ 19 ].

The concept of the child’s interests arises frequently in these publications. Pursuing or protecting these interests generally combines concern about the consequences of non-vaccination for the child with concern for autonomy, in the broad sense of being able to direct one’s life in accordance with one’s values or aims. Authors write about the interests of the child in both a general sense (i.e. the interests of the child outside of a legal context) and in a legal sense (the formal ‘best interests of the child standard’). The legal construction is used both to support (31, 6, 93) and to oppose vaccine refusal. Arguments that receiving a vaccine is in the legal ‘best interests of the child’ (21,39) posit that any deviation from a widely accepted legal view of the interests of a child should weigh the risk of harm to the child (68) irrespective of the parent’s beliefs (78), or that non-vaccination constitutes negligence or child endangerment [ 28 ]. On the other hand, some authors argue that, from a legal perspective, parents have the right to consent to or refuse vaccination ostensibly using the ‘child’s best interests standard’(93) and that there is insufficient legal precedent to argue that non-vaccination constitutes medical neglect [ 6 ].

Arguing from distrust and uncertainty

As previously noted, the sample included a set of papers written from the perspective of non-vaccinating parents. Most of these contributions seek to justify vaccine refusal, and many justifications were grounded in distrust. They call into question vaccine safety and effectiveness [ 12 , 13 , 14 , 18 ], and the accuracy of the reporting of adverse events following immunization (96). They claim financial conflicts, constructing clinicians, clinical medicine, and/or regulatory agencies as untrustworthy or non-credible [ 12 , 14 , 16 ]. They cite empirical studies of non-vaccinators to support parental preferences for natural infection over a vaccine (97). Non-vaccinating parents were not the only authors to make arguments in this vein. Some other authors cite the lack of absolute certainty of vaccine safety as justification for parents refusing vaccines in the interests of their children (28,76), especially regarding newer vaccines for which efficacy is not well-established (34). This line of argument depicts vaccine proponents as driven by commercial interests, thus justifying parental mistrust and refusal (34). Contra this, one paper asserts that refusal on the grounds of mistrust of government or medicine is not justifiable, as it is inconsistent with the scientific evidence and the well-established regulatory processes in place, such as the rigorous clinical testing required to develop and approve vaccines, and the systems established to report adverse events and ensure safety [ 8 ].

‘Response’ arguments: claims regarding the justifiability of different responses to non-vaccination

The literature examines four main responses to non-vaccination (i) government mandate policies (such as legal ramifications for refusing vaccination and vaccination as a school entry requirement), and other coercive policies, (ii) exemptions to mandate policies, (iii) individual practitioner and medical practice responses (including patient dismissal from practice for vaccine refusal, vaccinating against parents’ will, and nudging), and (iv) withholding health resources. The literature includes authors who argue that these responses are justifiable and others who argue that they are not. Much like the refusal arguments, some response arguments are absolute in their position, while others advocate weighing competing values in a context -specific way. Like refusal arguments, most arguments for and against particular responses to non-vaccinating parents draw from respect for autonomy, the doctrine of informed consent and the value of liberty, as well as considering consequences for the child and others. Other concepts appearing in these arguments include inequity, and the duties of governments and practitioners. Figure  7 represents proportionally the ’response’ arguments made in the reviewed literature.

‘Response’ arguments made in the literature on the ethics of vaccine refusal

As in the literature on refusal, many arguments about policy or practice responses to non-vaccinating parents depend on the interrelated concepts of respect for autonomy, informed consent and liberty. Five papers engage with the issue of practitioners vaccinating against parents’ will with respect to these concepts. They argue that forced vaccination by healthcare providers violates parents’ autonomy and/or the ethical requirement for informed consent, because vaccination carries risks (80,119), and clinicians have legal obligations to obtain valid consent for procedures (94). Some authors propose alternatives to forced vaccination, including focusing on rebuilding trust (rather than violating negative liberty) (32), and accepting that views on vaccination derive from plural and culturally-specific values [ 29 ]. On the other hand, proponents of forced vaccination do not engage with these concepts, instead deploying the harm principle and the legal ‘best interests of the child standard’ to justify their position. We explore this argument in the following section “Consequences for the child and others”.

Another set of papers make arguments about vaccine mandates that also draw on autonomy or liberty justifications, often weighing these against harm or risk of harm. Arguments justifying mandates are often legal in nature and use, for example, the harm principle or case law to argue that the freedom or liberty to choose not to vaccinate is limited by the risk of ill health and/or death to the child or others in the community, including vulnerable persons (83,91). One author argues that legal actions should be brought against those who harm others by refusing vaccination, as this would both discourage refusal and, in the case of any successful claims, compensate victims (55). Some authors argue that mandates are justifiable if the exercise of liberty rights poses a threat to public health (53,82,83,91,119). While those arguing that mandates are not justifiable sometimes rely on arguments about risk of harm—i.e. that in a low-incidence (and therefore low-risk) setting mandates cannot be justified (45, 87,104)—most make their arguments from autonomy, informed consent, and personal liberty and do not weigh these against the potential for harm (12,16,61,82,89,107,114). One author argues that even if mandates improve vaccination rates, they damage trust with parents and make refusers more steadfast in their decision (121), so are not sustainable. Finally, some authors present middle-ground positions that—in their view—are more autonomy- or liberty-preserving, including persuasion (121) or weakly enforced mandates (71), or argue that policy responses should take the least coercive approach that is feasible and effective to balance the needs of the individual with public health (117).

Those supporting conscientious objection to mandates argue that such provisions contribute to the collective good of a culture of respect for autonomy (82), or reflect the “American ideal” of personal freedom (66). Contra this, those opposed to conscientious objection provisions argue that challenges to mandates based in religious freedom have failed in case law, as the right to practice religion freely does not include the liberty to expose children or communities to disease (20,92). One author provides a qualified view of conscientious objection on religious grounds, arguing that such liberties could be justified only while high vaccination rates are maintained (109).

Authors disagree about whether certain policy or practice responses do, or do not, respect autonomy or uphold important liberties. For example, authors disagree on the effect of both nudges and conscientious objection policies on parental autonomy or liberty. With respect to nudges, some argue they are autonomy-preserving because they steer parents in a certain direction while allowing choice (106), do not override or challenge the strong views of deeply opposed opponents (42, 44) and uphold informed consent (121). Some supporters of nudging weigh multiple normative considerations, arguing that nudges that appeal to social responsibilities in a medical practice setting are justified because they appropriately balance parental autonomy against the practitioner’s responsibility to promote trust and collective benefits (3,80). Those opposed to nudges for vaccination decisions argue that the invasive nature of immunization increases the need for independent and informed decision making (60,113). These authors argue against a presumptive consultation style in general practice, proposing participatory clinical encounters (114), and using persuasion (42), as alternatives to more coercive approaches.

Consequences for the child and others

Many of the arguments in this literature consider individual and collective consequences—benefits, harms, burdens, and costs to society — and propose that these may override other normative considerations. The risk and prevention of harm is particularly pertinent here. For example, a parental decision can be overruled in cases where there is a significant risk of harm to the child (78), or nudges become more justifiable when the risk of harm to others is higher (3, 75).

Arguments about mandates often include concern about consequences, since it is inherent in a vaccine mandate that there will be some costs associated with non-vaccination. Mandate proponents argue that mandates ensure high vaccination rates, thus preventing disease outbreaks (39) and associated harms (97), so are in the best interest of individual children (28, 73, 111) and serve the greater good (4,28,73,79). Some justify mandates by proposing a duty to contribute to herd immunity, including under the “clean hands principle”, that is, an obligation not to participate in collectively harmful activities [ 1 , 5 ]. Conversely, some authors argue that mandates are not necessary to achieve high levels of population immunity, so state coercion is unjustified at a collective level or at the level of the individual child because each child receives limited benefit (94). Those opposing mandates also argue that vaccine safety is not absolute (88) and that mandates are a disutility, carrying associated costs with surveillance and enforcement (95). Other authors sought to balance these kinds of consequences against other normative considerations with respect to mandates, including the level of herd immunity, the risks of non-vaccination to the child and/or society, and respect for parental autonomy (32,53,88,119). One author argues that mandates protect ‘victims’ of the anti-vaccination movement from harms so long as certain conditions are met (43): that the vaccine can prevent infection and transmission, that individuals minimize their risk of exposure, and that the right of self-defense is preserved (e.g. in the case of allergy to vaccines).

Consequences are also important to arguments about conscientious objection, but here it is generally concerns about the impact on the collective. Some argue that exemptions should not be allowed because they may increase rates of disease or undermine individual or community health (20, 87, 118); others argue that if disease risk is low, exemptions are justified because those few individuals with exemptions do not pose a risk to others or herd immunity (20, 82, 105).

Consequences to the child and others are used to justify whether responses should be applied in general practice settings. As mentioned in the previous section, some authors justify healthcare workers vaccinating against a parent’s will using both the harm principle (69) and the legal ‘best interests of the child standard’ [ 25 ]; others suggest it is against the legal best interests of an older child to be forcibly vaccinated, as this may have a more detrimental impact than being unvaccinated (25,51). The best interests of the child are also invoked extensively to argue that non-vaccinating families should not be dismissed from medical practices (98,104, 26, 75). Here authors note that an unvaccinated child is more vulnerable to vaccine preventable diseases (9, 49), practice dismissal limits opportunities to access health care (31,52, 56,79,116) and the increased risk of harm from vaccine preventable diseases is transferred to other practices (9,47,49). One paper makes an argument about the consequences of treating non-vaccinating families for general practitioners, suggesting that practices caring for unvaccinated children should disclose this to other patients to minimize medicolegal risks, and should receive legal protection to account for the increased liability and risk of caring for these patients (40).

A small body of literature employs claims about who is responsible for the consequences of non-vaccination to make arguments about responses to non-vaccination. For example, one article seeks to justify discriminating against unvaccinated children with a vaccine preventable disease by limiting their access to health resources, relying on precedents such as coronary bypass surgery being withheld from obese people and smokers, and arguing that those who contribute to their own ill-health (in this case by not vaccinating) do not deserve healthcare (80). A related argument focuses on managing refugee camps during outbreaks that pose a direct and imminent threat of harm, proposing that the state is justified in withholding humanitarian aid from non-vaccinating refugees because the state is responsible for setting conditions that provide protection to (or prevent harm to) aid givers and public health [ 30 ].

Some critiques of policy or practice responses to non-vaccination emphasise that these responses can have inequitable effects and argue that this is unjustifiable. Exemption policies are a key focus here. Five papers argue against exemptions to vaccine mandates on the grounds that these unevenly distribute the risks and benefits of vaccinations (27,61,66, 73,118). These authors propose that the inaction of a few compromises the health of the most vulnerable community members (118) and disenfranchises those with medical contraindications for vaccines [ 27 ]. One author particularly focuses on home-schooled children, arguing that exempting them from vaccine mandates exposes both those children and society to harm, and that it is in the interests of these children and society that they be protected through vaccination (73). Some authors suggest that policy exemptions could be made justifiable by imposing conditions that offset potential inequities. On this view, exemptions could be justified so long as the refuser is prepared to make a financial or other contribution to help offset the potential financial burden of the diseases they may cause, or to otherwise contribute to social good [ 2 , 22 ].

Similarly, some opponents of coercive mandates or practice dismissal for non-vaccination critique these responses for having inequitable effects. It is argued that coercion risks creating a group of disenfranchised people (113) and that different people have different capacities to resist coercive policies (114). Similarly, dismissal leaves vulnerable children without advocacy (64), leads to patients not being treated equally (63) and marginalizes children from health care (74). One paper argues that family dismissal should be strongly discouraged, and an alternative mutually beneficial solution sought after considering the interests of the patient, physician, family, community, and society at large (74).

The duty of practitioners and the state

Some papers address the duties of practitioners and the duties of the state to respond to non-vaccination, in ways that go beyond simply weighing up consequences, implications for autonomy or freedom, or equity of impacts.

A variety of duties of practitioners are proposed. The first of these is to protect a child from their parent’s beliefs if those beliefs are likely to cause significant harm, which is used to justify initiating child protection proceedings to vaccinate against a parent’s will (67). Another is to protect patients in the waiting room from the risks posed by non-vaccinating patients, which is used to justify dismissing non-vaccinating patients from practice (9,26,38, 40,45). Counter-obligations are used to argue against practice dismissal. These include a health professional’s obligation to provide healthcare in the best interest of the child despite the parent’s decisions, and to deal with infectious disease as a part of their role (9,26,45,47, 56,101). Authors also argue that physicians’ obligations exclude enforcing parental accountability through dismissal, especially if that means the child is held accountable for the actions of their parents (47), and that continuing to provide care to a non-vaccinating family does not make the physician complicit in their decision (116).

It is sometimes asserted that the state is obliged to discourage non-vaccination on a number of grounds. This includes a fundamental duty of states to protect society [ 21 ], a responsibility of states to protect herd immunity as a common good or to reduce social and financial burdens and costs (53,119), and the state’s role to protect the common good in the face of risks to public health and the fallibility of individuals’ risk perception (54). Some of these arguments focus on exemptions from mandatory vaccination policies, proposing that states can not justify such exemptions because the government’s interest in protecting society outweighs the individual’s interest [ 21 ] or because vaccination is a social and moral good owed by a society to its children (118).

This review systematically explored and characterised the normative arguments made about parental refusal of routine childhood vaccination. Included publications addressed two types of arguments (i) ‘Refusal’ arguments (whether vaccine refusal is justified) and (ii) ‘Response’ arguments (whether various policy or practice responses to those who reject vaccines are justified). There were more ‘response’ arguments than ‘refusal’ arguments in the literature. On balance, most of the literature on ‘refusal’ arguments contended that it is not justifiable for parents to refuse vaccination for their children. Most of the ‘response’ argument literature argued against the various responses to non-vaccination put forward. However, compared to ‘refusal’ arguments, ‘response’ arguments were more varied and nuanced, and often came with caveats (e.g. exemptions to mandates are permissible if the disease burden is low).

The included articles predominantly originated from medical journals: these accounted for most of the papers focused on ‘response’ arguments. This may arise from the broader distribution of academic literature – there are more papers published in medicine than in the other disciplines represented in this review. It may also reflect the needs of readers of medical literature for guidance on how they should respond to non-vaccinating parents, highlighting the importance of making literature addressing the ethical dimensions of vaccine refusal accessible to immunization practitioners. Although there were some interdisciplinary perspectives, the dominance of the medical literature relating to ‘response’ arguments suggests that knowledge in this field may be advanced by incorporating more voices with expertise in ethics, law, and policy. This is especially important for deciding how to implement policy and practice responses to non-vaccination.

‘Refusal’ arguments were more common in the comparatively smaller collection of ethics/philosophy literature identified by this search, which may be, in part, a product of the differences in disciplinary traditions. While ethics/philosophy texts explore counterarguments and reach conclusions that are nuanced, and often with caveats, medical disciplines are primarily guided by practical considerations and a tradition of arguing from evidence rather than from ethical or philosophical principles. This privileging of evidence over principles may make it difficult to explore differing vaccination positions within the medical arena, potentially contributing to the adversarial clinical immunisation encounters described by vaccine-refusing parents and clinicians alike [ 7 , 18 , 19 ]. This pattern needs attention if ethical arguments are to have an impact in practice. As shown, most ethical arguments pay attention to evidence, as most ethical arguments include consequences in some way (see below). Ethical arguments can add nuance to biomedical thinking about consequences (e.g. consequences for individuals vs. the collective) and also about competing values (e.g. balancing consequences against concerns regarding autonomy, consent and liberty). The challenge for ethicists is to provide these arguments in an accessible and compelling form.

In fact, (i) consequences for the child and others, and (ii) respect for autonomy, the doctrine of informed consent and the value of liberty were dominant themes in both ‘refusal’ and ‘response’ arguments. Arguments were guided by common concepts such as the value of herd immunity, the prospect of harm to the child or others in the community and legal perspectives and precedents. The normative significance of parental trust, distrust, and uncertainty was a consideration unique to the ‘refusal’ arguments literature, driven in part by the five parental accounts from the special issue of Narrative Inquiry in Bioethics included in our sample. The concepts of inequity, and the duties of governments and practitioners only appeared in ‘response’ arguments. This is unsurprising: it reflects the purpose and perspective of these writers. An analysis of policy options is often required to bring inequity into view, and both clinicians and policymakers have obligations by virtue of their roles that can inform thinking about the right thing to do.

Many of the arguments justifying vaccine refusal aligned with the wider literature on the perspectives of non-vaccinating parents who value the freedom to make health decisions as caregivers, in what they perceive to be the best interest of their children [ 20 , 21 ]. These decisions are often based on doubts about vaccine safety or efficacy and are commonly initiated by a negative experience [ 19 , 20 , 22 ]. Unsurprisingly, arguments against rejecting childhood vaccines reflected the broader literature on how vaccine-supporting people view non-vaccination— including views that non-vaccinators are misinformed and disrupt social order, and that their actions are not based on reason or shared social values [ 23 ]. Common negative descriptors such as “anti-vaxxer” have similar valence in social discourse [ 24 ]. Those writing about vaccination should be aware of the potential for stigmatization and “othering” that can result by framing non-vaccination as a failure of parents [ 25 ]. When such arguments are used to inform policy and practice responses to non-vaccination, it introduces the potential for negative psychosocial impacts and further alienation of non-vaccinating parents.

Most ‘response’ arguments dealt with the justifiability of mandates and coercive policy. Generally, authors in favour of mandates prioritised the good of society; those against mandates prioritised individual choice. The large number of papers we found on mandates is unsurprising, given that these policies have been contentious. In Australia, federal and most state governments have mandates that require children to be vaccinated to be enrolled in childcare and for their families to be eligible for government financial assistance [ 26 ] Key political, academic and industry stakeholders argue that these mandates are designed to increase vaccination rates for the benefit of society [ 27 ]. On the other hand, Australian non-vaccinating parents express a belief that their children do not pose a threat to society, that all children should be treated in the same way, and that all parents should be able to make decisions for their children, regardless of vaccination status [ 28 ]. These perceptions of policy makers and non-vaccinating parents broadly represent the opposing arguments about mandates presented in this review. Facilitating a middle-ground approach to policy implementation may require closer attention to the values underlying these opposing views, and using a procedurally just approach to weigh them against one another.

In the context of an increasing number of systematic reviews in the field of bioethics, there has been recent criticism emerging about the use of these methods in bioethics. For example, Birchley and Ives (2022) argue that such methods are designed and therefore better suited to aggregation of quantitative data and not the complex and subjective nature of bioethical concepts and the theory-generating and interpretive approaches they require [ 29 ]. We argue that our application of the framework systematic review method - one of many well-established methods for systematic review and synthesis of qualitative and conceptual data - is appropriate for this research question and the application of our findings. Vaccine policy and practice requires a synthesis of what is known on relevant issues, and a systematic approach such as that used here provides a useful summary of the breadth of relevant ethical issues in a format that is accessible to policymakers. Our review has some limitations. Our aim was to map the range of normative arguments about vaccination refusal and policy. We did not have scope to present a novel ethical argument in response to our findings; this is an aim for future empirical and theoretical research. Most of the included literature focuses on high-income settings, predominantly the United States and the United Kingdom. In low-income settings, health services are often harder to access and levels of and reasons for vaccine rejection also differ in these settings. For example, political and cultural factors have been implicated in polio vaccine rejection in Nigeria [ 30 ], while low literacy, unemployment, and owning a mobile phone have been associated with polio vaccine refusal in Pakistan [ 31 ]. Our sampling period included a special issue of Narrative Enquiry in Bioethics which published narratives written by parents to communicate their normative positions on vaccination. These were mostly written by non-vaccinating parents and made up over one third of all arguments in the literature that support refusal. This is a strength in that it expanded the range of views represented in the review. However, it is also a limitation in that if this special issue had not been published within our sampling period, the range of arguments would have been more strongly skewed against vaccine refusal. These papers artificially increased the proportion of arguments in the scholarly domain that argue for vaccine refusal. It is a strength of our methodology that we were able to identify the unique perspective from which they were written and position them separately in our literature synthesis so that our representation of the literature distribution is not artificially skewed.

This review highlights an opportunity for interdisciplinary collaboration to widen the scope and reach of normative arguments about non-vaccination. Such collaboration can facilitate a broader understanding of and engagement with the ethical issues that may be relevant for practitioners, policymakers, and researchers in deciding how to respond to non-vaccinating parents. Arguments about the justifiability of non-vaccination and what should be done about it have the potential to positively influence routine childhood vaccination rates but can also alienate non-vaccinating families if not deployed with their perspectives in mind. There is an avenue for future work to further understand the influence of cultural context on normative arguments, especially within low- and middle-income settings. Moreover, there is an opportunity to further explore the influence and translation of scholarly ethical arguments into policy and practice responses to childhood non-vaccination.

Data Availability

The datasets generated and/or analysed during the current review are not publicly available, however the search terms used to generate the dataset are included in this published article.

Spier RE. Perception of risk of vaccine adverse events: a historical perspective. Vaccine. 2001;20:78–S84. https://doi.org/10.1016/S0264-410X(01)00306-1

Article   Google Scholar  

Nandi A, Shet A. Why vaccines matter: understanding the broader health, economic, and child development benefits of routine vaccination. Hum Vaccines Immunotherapeutics. 2020;16(8):1900–04. https://doi.org/10.1080/21645515.2019.1708669

Siddiqui M, Salmon DA, Omer SB. Epidemiology of vaccine hesitancy in the United States. Hum Vaccines Immunotherapeutics. 2013;9(12):2643–48. https://doi.org/10.4161/hv.27243

Beard FH, Hull BP, Leask J, et al. Trends and patterns in vaccination objection, Australia, 2002–2013. Med J Aust. 2016;204(7):275–75.

Ward PR, Attwell K, Meyer SB, et al. Understanding the perceived logic of care by vaccine-hesitant and vaccine-refusing parents: a qualitative study in Australia. PLoS ONE. 2017;12(10):e0185955.

Ward PR, Attwell K, Meyer SB, et al. Risk, responsibility and negative responses: a qualitative study of parental trust in childhood vaccinations. J Risk Res. 2017;1–14. https://doi.org/10.1080/13669877.2017.1391318

Wiley KE, Leask J, Attwell K, et al. Parenting and the vaccine refusal process: a new explanation of the relationship between lifestyle and vaccination trajectories. Soc Sci Med. 2020;263:113259.

Yaqub O, Castle-Clarke S, Sevdalis N, et al. Attitudes to vaccination: a critical review. Soc Sci Med. 2014;112:1–11.

Brewer NT, Chapman GB, Rothman AJ, et al. Increasing vaccination: putting psychological science into action. Psychol Sci Public Interest. 2017;18(3):149–207.

Attwell K, Navin MC, Lopalco PL, et al. Recent vaccine mandates in the United States, Europe and Australia: a comparative study. Vaccine. 2018;36(48):7377–84.

Attwell K, Seth R, Beard F et al. Financial interventions to increase vaccine coverage. Pediatrics 2020;146(6).

Attwell K, Navin M. How policymakers employ ethical frames to design and introduce new policies: the case of childhood vaccine mandates in Australia. Policy & Politics. 2022;50(4):526–47. https://doi.org/10.1332/030557321x16476002878591

Mertz M, Strech D, Kahrass H. What methods do reviews of normative ethics literature use for search, selection, analysis, and synthesis? In-depth results from a systematic review of reviews. Syst Rev. 2017;6(1):261. https://doi.org/10.1186/s13643-017-0661-x . [published Online First: 20171219].

Australian Bureau of Statistics., 2008, Australian and New Zealand Standard Research Classification (ANZSRC), availabe at: https://www.abs.gov.au/ausstats/[email protected]/0/4ae1b46ae2048a28ca25741800044242?opendocument

Kahrass H, Borry P, Gastmans C et al. PRISMA-Ethics – Reporting Guideline for Systematic Reviews on Ethics Literature: development, explanations and examples 2021 [Available from: OSF Preprints at https://osf.io/g5kfb

McDougall RJ, Notini L. Overriding parents’ medical decisions for their children: a systematic review of normative literature. J Med Ethics. 2014;40(7):448–52. https://doi.org/10.1136/medethics-2013-101446

Brunton G, Oliver S, Thomas J. Innovations in framework synthesis as a systematic review method. Res Synthesis Methods. 2020;11(3):316–30. https://doi.org/10.1002/jrsm.1399

Berry NJ, Henry A, Danchin M, et al. When parents won’t vaccinate their children: a qualitative investigation of Australian primary care providers’ experiences. BMC Pediatr. 2017;17(1):19. https://doi.org/10.1186/s12887-017-0783-2

Helps C, Leask J, Barclay L, et al. Understanding non-vaccinating parents’ views to inform and improve clinical encounters: a qualitative study in an Australian community. BMJ Open. 2019;9(5):e026299. https://doi.org/10.1136/bmjopen-2018-026299

Wiley KE, Leask J, Attwell K, et al. Parenting and the vaccine refusal process: a new explanation of the relationship between lifestyle and vaccination trajectories. Soc Sci Med. 2020;263:113259. https://doi.org/10.1016/j.socscimed.2020.113259

Attwell K, Leask J, Meyer SB, et al. Vaccine rejecting parents’ Engagement with Expert systems that inform Vaccination Programs. J Bioeth Inq. 2017;14(1):65–76. https://doi.org/10.1007/s11673-016-9756-7 . [published Online First: 20161201].

Christou-Ergos M, Leask J, Wiley KE. How the experience of medical trauma shapes Australian non-vaccinating parents’ vaccine refusal for their children: a qualitative exploration. SSM - Qualitative Research in Health. 2022;2:100143. https://doi.org/10.1016/j.ssmqr.2022.100143

Rozbroj T, Lyons A, Lucke J. The mad leading the blind: perceptions of the vaccine-refusal movement among australians who support vaccination. Vaccine. 2019;37(40):5986–93. https://doi.org/10.1016/j.vaccine.2019.08.023 . [published Online First: 20190823].

Court J, Carter SM, Attwell K, et al. Labels matter: use and non-use of ‘anti-vax’ framing in Australian media discourse 2008–2018. Soc Sci Med. 2021;291:114502. https://doi.org/10.1016/j.socscimed.2021.114502

Wiley KE, Leask J, Attwell K, et al. Stigmatized for standing up for my child: a qualitative study of non-vaccinating parents in Australia. SSM Popul Health. 2021;16:100926. https://doi.org/10.1016/j.ssmph.2021.100926 . [published Online First: 20210916].

Attwell K, Drislane S. Australia’s ‘No jab no play’ policies: history, design and rationales. Aust N Z J Public Health. 2022;46(5):640–46. https://doi.org/10.1111/1753-6405.13289

Attwell K, Navin M. How policymakers employ ethical frames to design and introduce new policies: the case of childhood vaccine mandates in Australia. Policy and Politics. 2022;1–22. https://doi.org/10.1332/030557321X16476002878591

Wiley K, Robinson P, Degeling C, et al. Get your own house in order’: qualitative dialogue groups with nonvaccinating parents on how Measles outbreaks in their community should be managed. Health Expect. 2022;25(4):1678–90. https://doi.org/10.1111/hex.13511

Birchley G, Ives J. Fallacious, misleading and unhelpful: the case for removing ‘systematic review’ from bioethics nomenclature. Bioethics. 2022;36(6):635–47. https://doi.org/10.1111/bioe.13024

Yahya M. Polio vaccines—no thank you! Barriers to polio eradication in Northern Nigeria. Afr Affairs. 2007;106(423):185–204. https://doi.org/10.1093/afraf/adm016

Khattak FA, Rehman K, Shahzad M, et al. Prevalence of parental refusal rate and its associated factors in routine immunization by using WHO Vaccine Hesitancy tool: a Cross sectional study at district Bannu, KP, Pakistan. Int J Infect Dis. 2021;104:117–24. https://doi.org/10.1016/j.ijid.2020.12.029

Download references

This review was funded by the Australian National Health and Medical Research Council, grant number GNT1126543.

Author information

Authors and affiliations.

Sydney School of Public Health, The University of Sydney, Edward Ford Building A27, Sydney, 2006, Australia

Kerrie Wiley, Maria Christou-Ergos, Penelope Robinson & Catherine Helps

Australian Centre for Health Engagement, Evidence and Values, The University of Wollongong, Wollongong, 2522, Australia

Chris Degeling & Stacy M Carter

Melbourne School of Population and Global Health, The University of Melbourne, Melbourne, 3010, Australia

Rosalind McDougall

School of Social Sciences, Asian Studies & Politics, International Relations, University of Western Australia, Perth, 6009, Australia

Katie Attwell & Shevaun Drislane

You can also search for this author in PubMed   Google Scholar

Contributions

KW contributed to study design and search strategy development, ran the searches, managed the screening and inclusion process, screened articles for inclusion, extracted data, analysed and interpreted data and co-led manuscript drafting; MC ran updated searches, screened articles for inclusion and extracted data, assisted with analysis and interpretation and co-led manuscript drafting; CD contributed to study design and search strategy development, provided technical guidance, screened articles for inclusion and contributed to manuscript drafts; RM contributed to study design and search strategy development, provided technical guidance, screened articles for inclusion and contributed to manuscript drafts; PR screened articles for inclusion, extracted data, assisted with analysis and contributed to manuscript drafts; KA contributed to search strategy development, screened articles for inclusion and contributed to manuscript drafts; CH screened articles for inclusion and contributed to manuscript drafts; SD screened articles for inclusion and contributed to manuscript drafts; SMC contributed to study design and search strategy development, provided technical guidance, screened articles for inclusion and contributed to manuscript drafts.

Corresponding author

Correspondence to Kerrie Wiley .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Cite this article.

Wiley, K., Christou-Ergos, M., Degeling, C. et al. Childhood vaccine refusal and what to do about it: a systematic review of the ethical literature. BMC Med Ethics 24 , 96 (2023). https://doi.org/10.1186/s12910-023-00978-x

Download citation

Received : 20 February 2023

Accepted : 31 October 2023

Published : 08 November 2023

DOI : https://doi.org/10.1186/s12910-023-00978-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Vaccination
• Immunization
• Vaccine refusal
• Systematic review
• Normative literature
• Medical ethics

BMC Medical Ethics

ISSN: 1472-6939

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• News & Views
• Burden of proof:...

Burden of proof: combating inaccurate citation in biomedical literature

• Related content
• Peer review
• Nicholas Peoples , MD student 1 ,
• Truls Østbye , vice chair (research) and professor 2 ,
• Lijing L Yan , professor and head of non-communicable disease research 3
• 1 Baylor College of Medicine, Houston, TX, USA
• 2 Family Medicine and Community Health, Duke University, Durham, NC, USA
• 3 Global Health Research Center, Duke Kunshan University, Kunshan, Jiangsu Province, China
• Correspondence to: [email protected]

Inappropriate, misleading, missing, and inaccurate citations pervade the biomedical literature. Nicholas Peoples and colleagues argue that new strategies can better enable scientific references to function as an accurate web of knowledge

Key messages

Up to 25% of all citations in the general scientific literature are inaccurate and mislead physicians, academics, and policy makers

The emergence of artificial intelligence (AI) powered large language models such as ChatGPT has the potential to both enable and mitigate inaccurate citation on a scale not previously possible

Researchers need new strategies to ensure that scientific references function as an accurate web of knowledge

We make the case that peer reviewed journals consider adopting a required statement on the integrity of cited literature, using the adoption of required conflict of interest statements as a proof of concept

Even without a name, it is a devil we all know: an article cites a source that does not support the statement in question, or, more commonly, the initial reference sends the reader down a rabbit hole of references, the bottom of which is difficult to find and interpret. This causes two problems. Firstly, it may propagate data that are false, misinterpreted, or both, spurring “academic urban legends” that become circulated as truth. 1 This delays true results from reaching the literature and allows incorrect ideas to masquerade as facts. Second, it undermines respect for the process of literature review, effacing the foundation of good scientific inquiry into a mere box ticking exercise. This cheapens the value of background and discussion sections in scholarly articles and encourages trainees and young investigators to practise sloppy research.

These errors might be especially problematic for doctors and the general public, “who are not focused on the scientific study of a narrow research topic and thus are less prone to identify rhetorically misleading statements or outright factual errors.” 2 Leung and colleagues, for example, document clear patterns of inaccurate citation that misrepresent the conclusions of a single paragraph statement in the New England Journal of Medicine on the safety of opiate use. 3 4 They argue that these misrepresentations might have contributed to the North American opioid crisis “by helping to shape a narrative that allayed prescribers’ concerns about the risk of addiction associated with long term opioid therapy.” 3

Recent estimates indicate citation error rates of 11-15% in the biomedical literature 2 5 and up to 25% in the general science literature. 6 In a review of 4912 citations, 38.4% of these errors were citing non-existent findings, 15.4% were incorrect interpretations of findings, and 20% were chains of inaccurate citations copied forward from paper to paper. 5 This indicates that mis-citation is widespread. A surgical study 7 was found to be misquoted by 40% of the articles that cited it, 8 creating an unsupported but widely accepted guideline for how an orthopaedic procedure should be performed. This shows that mis-citation might also deeply mischaracterise individual scientific works. Finally, to understand how an entire scientific belief system might evolve, a 2009 study systematically mapped out the full citation chain for a particular scientific claim related to amyloid β. Among its findings was the “marked expansion of the belief system by papers presenting no data addressing it; and forms of invention such as the conversion of hypothesis into fact through citation alone.” 9 So, improper citation can even credibly distort the scientific consensus.

Rekdal offers granular insight into this process in his excellent analysis of the “iron content in spinach” myth ( fig 1 ). 1 In a 1981 article entitled “Fake!”, 10 Terry Hamblin believed he was debunking an erroneous claim about the iron content of spinach but was unknowingly using incorrect information himself. Others then cited and transformed his ideas into even greater inaccuracies. 1 11 Rekdal convincingly makes the case that even later authors, such as Larsson in 1995, 12 borrowed the conclusions of articles that cited Hamblin without actually consulting the 1981 paper directly, further distorting the truth. In the end, Hamblin’s accidental rumour was only debunked in 2010 (some 30 years later), 13 and he tried in vain to extinguish it to up until his death in 2012. 1 14

Creation, propagation, and debunking of an academic urban legend

• Download figure
• Open in new tab
• Download powerpoint

This all makes for a poor report card, especially given that erroneous citation has been rampant since at least 1931 (and, judging by Shull’s exasperated remarks, with little progress in the time since). 15 But modern figures risk becoming underestimates: emerging artificial intelligence (AI) powered large language models now enable inaccurate citation with an efficiency and scale not previously possible. On a broad scale, academia is clearly grappling with how to reconcile technological progress and traditional research ethics with concerns that AI may “hallucinate” sources or fabricate data. 16 17 So why do we turn a blind eye if such fabrications are human, so long as they are neatly packaged into scholarly citations that look and sound the part? 18

Types of inadequate citation

Just as there is no universally accepted term—misquotation 8 versus quotation errors, 2 erroneous citation 15 versus inadequate citation, 12 and so on—there is no universally accepted classification scheme. Steven Greenberg developed a “vocabulary” for the “citation distortions” he encountered in his study of amyloid β, which offers one useful framework, but it primarily describes mechanisms and consequences of select forms of poor citation without exploring underlying causes, which also matter. 9 Others discuss “common citation errors” but do not report the methodology for how they arrived at these categories. 19 We propose that rigorous, systematic categorisation, a taxonomy and nomenclature of inadequate citation, or both, are important next steps. Box 1 provides a concise overview of some common documented types of mis-citation.

Selected examples of inadequate citation practices

Biased —preferentially citing certain sources, perhaps those of colleagues, because the author is simply more familiar with them, regardless of whether they are the best supporting works for the claim at hand 19

Coercion: pre-submission —senior or principal investigator incorrectly adds a reference to a work created by a trainee (such as a PhD student), who does not feel empowered to decline or challenge this move 20

Coercion: within submission —during peer review, a reviewer or editor instructs the author to cite publications co-authored by the reviewer or editor 20

Editing —a work is initially cited correctly, but as the draft is edited and sentences are eliminated or consolidated, perhaps by multiple authors, a correctly cited study inadvertently becomes misquoted 19

Hallucinatory —citing a source or conclusion that was convincingly fabricated by artificial intelligence without confirming existence of the source or veracity of the claims 16 17

“Lazy author” —adding citations without reading the work in full (or at all) 21 22 23

Misinterpretation —wrongly interpreting the source 2 19

Missing —making a statement without citing the source 19

Non-contextual —citing a part of the conclusion, misrepresenting the results 9 19

Plagiarism —representing others’ work as one’s own by omitting citations

Secondary —citing a secondary source (such as a review article) when a primary source exists 19 24

Self-citation —inappropriately citing one’s own work without sufficient justification

Temporal —Citing a work that was read, but some time ago, such that the author inaccurately recollects the findings 25

Strategies to mitigate mis-citation

There are few checks and balances on erroneous citation. The first line of defence (pre-submission quality control) is a standard expectation of academic integrity. In the age of “publish or perish,” however, competing interests, such as enormous pressure on investigators to exhibit constant productivity, can make it tempting to cut corners in pursuit of expediency. 6 21 24 Market forces are in tune to this. Consensus, for example, is a new, AI powered search engine designed specifically for academics ( https://consensus.app/search/ ). Given a search query, it will produce a list of 5-10 peer reviewed papers with a short synopsis of each. The more widely used ChatGPT has been found to fabricate both facts and legitimate sounding “scholarly sources” to provide a convincing—rather than factual—answer to a prompt. 16 17 Although these tools are multi-purpose, powerful, and certainly promising adjuvants for literature review, they also exponentially enable the ease and scale of producing inaccurate citations.

The second line of defence is peer review, which currently serves as the major safety net to catch mis-citations after submission. History has shown, however, that this can also be inefficient, inconsistent, and insufficient. 6 26 27 28 (And, as others point out, peer review may even encourage mis-citation.) 20 Moreover, the ultimate responsibility for auditing cited literature should not rest with peer reviewers, but with those who selected the literature to support their claims.

The third line of defence is post-publication review, which is often either inordinately difficult 29 30 or even directly opposed. 31 Although non-replicable primary results have been caught and overturned after publication, there is little drive to do the same for inaccurate citations. This effectively allows them to live on into eternity. Although we are not the first to highlight these problems, 15 32 we argue that prevailing strategies are insufficient.

New tools in our toolkit

Just as AI might exacerbate erroneous citation, we suggest it could also be part of the solution. It might, for example, eventually be possible for AI programmes to be integrated into manuscript submission portals to confirm that all cited works actually exist. Even more useful, however, will be AI modalities developed to assess citations for accuracy and to flag potential discrepancies for review. This would not replace human review, but something with reasonable sensitivity and high specificity could provide an initial screen that alerts reviewers to instances of potential mis-citation without creating additional work.

Another idea is for journals to ask authors to attest that an article contains no inaccurate citations ( box 2 ), much in the same way that they are expected to make a statement about conflicts of interest.

Example works cited statement

The author(s) certify that all works cited were read in full by at least one author at the time of writing this manuscript; are necessary to support the intellectual foundation of the work; were added without undue coercion; and do not reflect inappropriate self-citation. We affirm that this document cites primary sources whenever possible, transparently discloses the source of all secondary information presented, and accurately represents both the objective findings and earnest spirit of all works cited.

These ideals are implicit in any submission to a peer reviewed journal, so making them explicit presents no addition burden. The key question, then, is whether such declarations are useful. To answer this, we can compare older to more recent literature as journals have progressively adopted more rigorous standards. Specifically, conflict of interest (COI) statements are a useful analogy to our proposed works cited statement, because when there are no competing interests to declare, they function as an attestation. These statements became increasingly common as concerns grew about the influence of money and corporate interests in research. Estimates for the proportion of journals requiring a COI statement vary by discipline but consistently show a positive trend. Some commonly cited examples include 16% (220 of 1367 highly ranked scientific and biomedical journals) in 1997, 33 33% (28 of 84 journals from 12 scientific disciplines) in 2007, 34 89.7% (358 of 399 “high impact biomedical journals”) in 2011, 35 and 96% (224 of 227 public and occupational health journals) in 2016. 36 It comes as no surprise, then, that none of 47 trials on febrile neutropenia published from 1981 to 2000 contained a COI statement. 37 (Additionally, only 29 reported that informed consent was obtained and only 22 reported approval of a research ethics committee.) To see how this compares with more recent literature, we reviewed a random sample of 100 research papers published in 2022 in the New England Journal of Medicine , the Journal of the American Medical Association , and The BMJ , finding that 100% of articles included COI statements (and likewise, statements on institutional review board approval and informed consent, when applicable) (see supplementary file on bmj.com).

A COI statement encourages transparency because inadequate disclosure carries consequences. Similarly, a “works cited statement” ( box 2 ) might encourage diligence as it implies that references will be scrutinised, and mis-citation will be penalised. Assuming that most authors operate on good intent (striving for their work to be accurate) or self-interest (striving to avoid delays in publication), or both, the act of formal attestation might inspire greater diligence in crafting a reference list or double checking it for accuracy before submission. Even with inadequate disclosure, however, statements still offer important functionality. A 2014 phase 3 trial, for example, was publicly redacted when it was discovered that the authors lied in their COI statement. 38 Here, the COI statement created a clear mismatch between the authors’ words and deeds. This alerted a vigilant reader, provided the journal with irrefutable justification for corrective action, and, ultimately, halted the dissemination of untrustworthy information. Similarly, a works cited statement found to have inadequate or inaccurate disclosure might raise a red flag and lead reviewers or readers to scrutinise a work more closely, potentially to catch other important flaws. If the author’s formal assurances for something as basic as a reference list cannot be taken at face value, what else might they have been mistaken or misleading about?

COI statements enable new inquiries into research bias. Landmark studies have shown, for example, a strong association between pharmaceutical industry funding and likelihood of reporting significant results. 39 40 Another study analysed 767 clinical trials and found a strong association between failure to disclose informed consent and poor methodological quality. 41 A works cited statement, then, would enable researchers to ask similar questions about inaccurate citations, such as whether studies with an inaccurate works cited statement are associated with poor methodological quality or inflated results. Fact checking citation accuracy can be labour intensive, but there is a critical mass of people who do this sort of work already, reflected in the growing literature on mis-citation. 1 2 3 5 6 8 9 19 20 21 22 23 24 25 30 32

So, declarations do seem to have beneficial value. They are not a perfect, systemic failsafe, but they do provide authors, editors, peer reviewers, and journal readers with additional opportunities to promote essential quality control. Similar cases can be made for the required declaration of institutional review board approval, author contributions, informed consent, and emerging declarations of the role of AI. As thousands of erroneous citations now raise legitimate questions about inaccuracies in published research, there is a clear case for adopting similar safeguards in this arena as well. To make such a declaration meaningful, however, it must be enforced. 42

As a final strategy, we propose that journals add two internal questions for reviewers: were any improper citations noted during the review of this paper? And did you recommend the authors to cite any studies in which you are a coauthor or otherwise have a vested interest? If the answer to either question is yes, reviewers must provide specific details, which are sent to the editor. We propose that a submitted study with erroneous citations should be withdrawn from consideration if the errors are pervasive, mischaracterise the background or methods, inappropriately shape interpretation of the results, or otherwise betray a serious lack of expected due diligence. For manuscripts with less egregious mis-citation, we recommend that reviewers and editors still adopt a low threshold to mandate revisions, as “there is no good reason to allow . . . inexact and non-verifiable referencing to pervade scientific literature.” 6 We further propose that an editor should closely scrutinise situations in which a reviewer has recommended citation of their own works. 20 If already published when the errors are caught, authors should be asked to amend the work (without additional fees 29 ), which is eminently possible in the digital era. Critically, we extend investigators the benefit of the doubt: many inaccurate citations are simply the result of honest mistakes. Thus, this higher standard is primarily meant to deter carelessness and promote good practice. As is the case broadly in medicine, primary prevention is often the best policy.

Looking ahead

Mis-citation has heretofore been inadequately tackled. By acknowledging the fault lines in current practice, prioritising the development of a rigorous and standardised classification scheme for inadequate citation, and codifying accurate literature review into a routine pre-submission declaration, we can strive to better enshrine integrity into medical scholarship. We encourage journals to consider adopting a pre-submission declaration. It has the potential to deter inappropriate manuscript submissions and facilitate correction after publication, with little added cost or inconvenience. Over time, the higher standard might also instil greater respect among the next generation of young investigators for this fundamental pillar of scientific inquiry. Most importantly, if it yields improvement in overall citation quality, this will help the scientific literature better function as an accurate web of knowledge.

Acknowledgments

We thank Vianna Quach, Dianne Wade, and Alexandra Alvarez for expert editorial feedback on early drafts of this work.

Works cited statement: The authors certify that all works cited were read in full by at least one author at the time of writing this manuscript; are necessary to support the intellectual foundation of the work; were added without undue coercion; and do not reflect inappropriate self-citation. We affirm that this document cites primary sources whenever possible, transparently discloses the source of all secondary information presented, and accurately represents both the objective findings and earnest spirit of all works cited.

Funding: This work received no funding or financial support of any kind.

Contributors and sources: This work was conceived by NP, who wrote the first draft and acts as the guarantor for this work. LLY and TO provided critical review that helped shape the key intellectual output of this work. NP holds an MSc from Duke University in the US and is currently a 4th year MD student at Baylor College of Medicine and a Schwarzman Scholar at Tsinghua University. LLY is head of non-communicable disease research and tenured faculty at Duke Kunshan University in China. She has published over 150 studies in global health, many involving complex multi-country randomised controlled trials. TO is a physician and epidemiologist. He has published over 690 peer reviewed studies and holds professorships at Duke University, Duke Kunshan University, and Duke-NUS University in Singapore. This work is the product of three experienced, professionally and culturally diverse researchers who disdain corner cutting in research and want to advocate for integrity and quality in the biomedical sciences.

Patient involvement: No patients were involved in the creation of this manuscript.

Competing interests: We have read and understood BMJ policy on declaration of interests and have no interests to declare.

• Leung PTM ,
• Macdonald EM ,
• Stanbrook MB ,
• Dhalla IA ,
• Juurlink DNA
• Pavlovic V ,
• Weissgerber T ,
• Stanisavljevic D ,
• Smith N Jr . ,
• Cumberledge A
• Porrino JA Jr . ,
• Greenberg SA
• ↵ Skrabanek P, McCormick J. Follies and Fallacies in Medicine, 3rd edn. Tarragon Press, 1998.
• ↵ Sutton M. How the spinach, Popeye and iron decimal point error myth was finally busted. https://archive.ph/ASqr
• ↵ Hamblin TJ. Spinach—I was right for the wrong reason. Mutations of Mortality. 2010. http://mutated-unmuated.blogspot.com/2010/12/spinach-i-was-right-for-wrong-reason.html .
• Alkaissi H ,
• McFarlane SI
• ↵ Bohannon M. Lawyer used ChatGPT in court-and cited fake cases. A judge is considering sanctions. Forbes Magazine. 2023. https://www.forbes.com/sites/mollybohannon/2023/06/08/lawyer-used-chatgpt-in-court-and-cited-fake-cases-a-judge-is-considering-sanctions/?sh=1f047f3f7c7f
• Ingelfinger FJ
• Agarwal A ,
• Avidor-Reiss T ,
• Hamoda TAA ,
• Wilhite AW ,
• Simkin MV ,
• Roychowdhury VP
• Teixeira MC ,
• Thomaz SM ,
• Michelan TS ,
• ↵ Roig M. Avoiding plagiarism, self-plagiarism, and other questionable writing practices: a guide to ethical writing. Office of Research Integrity. 2015. https://ori.hhs.gov/avoiding-plagiarism-self-plagiarism-and-other-questionable-writing-practices-guide-ethical-writing
• Schroter S ,
• Carpenter J ,
• Allison DB ,
• George BJ ,
• Hosseini M ,
• Gordijn B ,
• Cooper DJ ,
• Gangadharan L ,
• de Lacey G ,
• Krimsky S ,
• Rothenberg LS
• Ancker JS ,
• Pericas JM ,
• Hernández C ,
• Resnik DB ,
• Konecny B ,
• Kissling GE
• Weingarten MA ,
• Leibovici L
• ↵ Notice of Retraction: Pak CS, et al. A Phase III, Randomized, Double-Blind, Matched-Pairs, Active-Controlled Clinical Trial and Preclinical Animal Study to Compare the Durability, Efficacy and Safety between Polynucleotide Filler and Hyaluronic Acid Filler in the Correction of Crow’s Feet: A New Concept of Regenerative Filler. J Korean Med Sci 2014; 29(Suppl 3): S201-S209 . J Korean Med Sci 2016 ; 31 : 330 - 330 . doi: 10.3346/jkms.2016.31.2.330 .   pmid: 26839493 OpenUrl CrossRef PubMed
• Lexchin J ,
• Djulbegovic B ,
• Friedman LS ,
• Ruiz-Canela M ,
• de Irala-Estevez J ,
• Martínez-González MA ,
• Gómez-Gracia E ,
• Fernández-Crehuet J
• ↵ Travaglia J, Hinchcliff R, Carter D, Billington L, Glennie M, Debono D. Attestation by governing bodies: literature review. Australian Commission on Safety and Quality in Health Care. https://www.safetyandquality.gov.au/publications-and-resources/resource-library/attestation-governing-bodies-literature-review