Criteria | OSA | MsR | Roles | Content | Context |
---|---|---|---|---|---|
Authorization strategy | DAC | MAC | Hybrid | Hybrid | Hybrid |
Granularity of control | L | L | M | H | H |
Least privilege principal support | L | M | M | H | O |
Dynamic authorization | ✗ | ✗ | ✓ | ✓ | ✓ |
Separation of duty | ✗ | O | ✓ | ✓ | ✓ |
Vulnerable to attacks | ✓ | ✓ | ✗ | ✗ | ✗ |
Bypass | ✗ | ✗ | ✗ | ✗ | ✓ |
Conflict resolution or prevention | ✗ | ✗ | ✗ | O | O |
Operational/situational awareness | ✗ | ✗ | ✓ | ✓ | ✓ |
Privileges/capabilities discovery | ✓ | ✗ | ✓ | O | ✗ |
a Except the MLS database model;
Depends on the underlying access control model;
Supported by the Chinese-Wall model;
Supported by ABAC and EBAC models; and
Supported by the VBAC model
Abadi , M. and Fournet , C. ( 2003 ), “ Access control based on execution history ”, NDSS , Vol. 3 , pp. 107 - 121 .
Abrams , M.D. ( 1995 ), “ Renewed understanding of access control policies ”, in Proceedings of the 16th National Computer Security Conference-Information System Security: User Choices , pp. 87 - 96 .
Ahmad , A. and Whitworth , B. ( 2011 ), “ Access control taxonomy for social networks ”, in 2011 7th International Conference on Information Assurance and Security . (Ed.) by Ieee Corporate Author . IEEE , pp. 256 - 261 , ISBN: 978-1-4577-2155-7 , doi: 10.1109/ISIAS.2011.6122829 .
Almehmadi , A. and El-Khatib , K. ( 2013 ), “ Authorized! access denied, unauthorized! access granted ”, in Proceedings of the 6th International Conference on Security of Information and Networks , pp. 363 - 367 .
Astrahan , M.M. , Blasgen , M.W. , Chamberlin , D.D. , Eswaran , K.P. , Gray , J.N. , Griffiths , P.P. , King , W.F. , Lorie , R.A. , McJones , P.R. , Mehl , J.W. , Putzolu , G.R. , Traiger , I.L. , Wade , B.W. and Watson , V. ( 1976 ), “ System R: relational approach to database management ”, ACM Transactions on Database Systems (TODS) , Vol. 1 No. 2 , pp. 97 - 137 .
Atlam , H.F. , Azad , M.A. , Alassafi , M.O. , Alshdadi , A.A. and Alenezi , A. ( 2020 ), “ Risk-based access control model: a systematic literature review ”, Future Internet , Vol. 12 No. 6 , p. 103 , doi: 10.3390/fi12060103 .
Banerjee , A. and Naumann , D.A. ( 2004 ), “ History-based access control and secure information flow ”, In International Workshop on Construction and Analysis of Safe, Secure, and Interoperable Smart Devices . Springer , pp. 27 - 48 .
Bartoletti , M. , Degano , P. and Ferrari , G.L. ( 2005 ), “ Historybased access control with local policies ”, in Sassone , V. (Ed.), Foundations of Software Science and Computational Structures , Springer Berlin Heidelberg , Berlin, Heidelberg , pp. 316 - 332 . ISBN: 978-3-540-31982-5 .
Bell , D.E. and La Padula , L.J. ( 1976 ), Secure computer system: unified exposition and multics interpretation , Tech. rep. Mitre Corp Bedford MA .
Benantar , M. ( 2005 ), Access Control Systems: security, Identity Management and Trust Models , Springer Science and Business Media .
Bertino , E. , Bettini , C. , Ferrari , E. and Samarati , P. ( 1996a ), “ A temporal access control mechanism for database systems ”, IEEE Transactions on Knowledge and Data Engineering , Vol. 8 No. 1 , pp. 67 - 80 .
Bertino , E. , Bettini , C. , Ferrari , E. and Samarati , P. ( 1996b ), “ Supporting periodic authorizations and temporal reasoning in database access control ”, VLDB , Citeseer , pp. 472 - 483 .
Bertino , E. , Samarati , P. and Jajodia , S. ( 1997 ), “ An extended authorization model for relational databases ”, IEEE Transactions on Knowledge and Data Engineering , Vol. 9 No. 1 , pp. 85 - 101 .
Bertino , E. , Bonatti , P.A. and Ferrari , E. ( 2000 ), “ TRBAC: a temporal role-based access control model ”, in Proceedings of the 5th ACM Workshop on Role-Based Access Control , pp. 21 - 30 .
Bertino , E. and Sandhu , R. ( 2005 ), “ Database security – concepts, approaches, and challenges ”, IEEE Transactions on Dependable and Secure Computing , Vol. 2 No. 1 , pp. 2 - 19 .
Bertino , E. , Ghinita , G. and Kamra , A. ( 2011 ), “ Access control for databases: concepts and systems ”, Foundations and Trends in Databases , Vol. 3 Nos 1/2 , pp. 1 - 148 , doi: 10.1561/1900000014 , ISSN: 1931-7883 .
Bertino , E. ( 2016 ), “ Data security and privacy: concepts, approaches, and research directions ”, in 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC) , IEEE , pp. 400 - 407 , ISBN: 978-1-4673-8845-0 , doi: 10.1109/COMPSAC.2016.89 .
Bertino , E. ( 1998 ), “ Data security ”, Data and Knowledge Engineering , Vol. 25 Nos 1/2 , pp. 199 - 216 .
Bogaerts , J. , Decat , M. , Lagaisse , B. and Joosen , W. ( 2015 ), “ Entity-based access control: supporting more expressive access control policies ”, In Proceedings of the 31st Annual Computer Security Applications Conference , pp. 291 - 300 .
Brewer , D.F. and Nash , M.J. ( 1989 ), “ The Chinese wall security policy ”, in IEEE symposium on security and privacy , Oakland ,Vol. 1989 , p. 206 .
Browder , K. and Davidson , M.A. ( 2002 ), “ The virtual private database in oracle9ir2 ”, In: Oracle Technical White Paper, Oracle Corporation 500.280 .
Center, Computer Security Resource ( 2022 ), “ Glossary ”, available at: https://csrc.nist.gov/glossary (accessed 14 April 2022 ).
Cheng , P.-C. , Rohatgi , P. , Keser , C. , Karger , P.A. , Wagner , G.M. and Reninger , A.S. ( 2007 ), “ Fuzzy multi-level security: an experiment on quantified risk-adaptive access control ”, in 2007 IEEE Symposium on Security and Privacy (SP’07) , IEEE , pp. 222 - 230 .
Clark , D.D. and Wilson , D.R. ( 1987 ), “ A comparison of commercial and military computer security policies ”, in 1987 IEEE Symposium on Security and Privacy , IEEE , pp. 184 - 184 .
Council, Information Technology Industry ( 2016 ), “ Information technology: next generation access control - generic operations and data structures (NGAC GOADS) ”, in American National Standard for Information Technology INCITS, 526-2016 .
Council, Information Technology Industry ( 2018 ), “ Information technology: next generation access control - functional architecture (NGAC-FA) ”, in American National Standard for Information Technology INCITS, 499-2018 .
Crampton , J. and Loizon , G. ( 2002 ), SARBAC: a new model for role-based administration . Tech. rep. Technical Report BBKCS-02-09 , Birkbeck College, University of London .
Cuppens , F. and Miège , A. ( 2003 ), “ Administration model for or-BAC ”, in Meersman , R. and Tari , Z. (Eds), On The Move to Meaningful Internet Systems 2003: OTM 2003 Workshops , Springer Berlin Heidelberg , Berlin, Heidelberg , pp. 754 - 768 , ISBN: 978-3-540-39962-9 .
Damiani , M.L. , Bertino , E. , Catania , B. and Perlasca , P. ( 2007 ), “ GEO-RBAC: a spatially aware RBAC ”, ACM Transactions on Information and System Security (TISSEC) , Vol. 10 No. 1 , p. 2–es .
Danwei , C. , Xiuli , H. and Xunyi , R. ( 2009 ), “ Access control of cloud service based on ucon ”, in IEEE International Conference on Cloud Computing , Springer , pp. 559 - 564 .
Eckert , C. ( 2014 ), IT-Sicherheit , 9th ed ., De Gruyter Oldenbourg , ISBN: 978-3-486-85916-4 .
Edjlali , G. , Acharya , A. and Chaudhary , V. ( 1998 ), “ History-based access control for mobile code ”, in Proceedings of the 5th ACM Conference on Computer and Communications Security , pp. 38 - 48 .
Ferraiolo , D.F. and Kuhn , D.R. ( 1992 ), “ Role-based access controls ”, in Proceedings of the 15th NIST-NSA National Computer Security Conference , pp. 554 - 563 .
Ferraiolo , D. , Cugini , J. and Kuhn , D.R. ( 1995 ), “ Role-based access control (RBAC): features and motivations ”, in Proceedings of the 11th annual computer security application conference , pp. 241 - 248 .
Ferraiolo , D. , Chandramouli , R. , Kuhn , R. and Hu , V. ( 2016 ), “ Extensible access control markup language (XACML) and next generation access control (NGAC) ”, in Proceedings of the 2016 ACM International Workshop on Attribute Based Access Control , pp. 13 - 24 .
Ferrari , E. ( 2009 ), “ Access control ”, in Tamer Ozsu , M. and Ling , L. (Eds), Encyclopedia of Database Systems , Springer , New York, NY , pp. 7 - 11 , doi: 10.1007/978-0-387-39940-9_6 , isbn: 978-0-387-35544-3 .
Fong , P.W. ( 2004 ), “ Access control by tracking shallow execution history ”, IEEE Symposium on Security and Privacy , IEEE , pp. 43 - 55 .
Fong , P.W. ( 2011 ), “ Relationship-based access control: protection model and policy language ”, in Proceedings of the first ACM conference on Data and application security and privacy , pp. 191 - 202 .
Gao , X. , Jiang , Z.M. and Jiang , R. ( 2013 ), “ A novel data access scheme in cloud computing ”, Advanced Materials Research , Vols 756/759 , pp. 2649 - 2654 .
Gates , C. ( 2007 ), “ Access control requirements for web 2.0 security and privacy ”, in IEEE Web 2.0 , pp. 12 - 15 .
Griffiths , P.P. and Wade , B.W. ( 1976 ), “ An authorization mechanism for a relational database system ”, ACM Transactions on Database Systems (TODS) , Vol. 1 No. 3 , pp. 242 - 255 .
Harris , S. ( 2012 ), CISSP All-in-One Exam Guide , 6th ed ., McGraw-Hill , New York, NY, Chicago, San Francisco, Lisbon, London, Madrid, Mexico City, Milan, New Delhi, San Juan, Seoul, Singapore, Sydney, Toronto , ISBN: 978-0-07-178173-2 .
Hosseini , A. and Azgomi , M.A. ( 2010 ), “ HRBAC: historical role-based access control ”.
Hota , C. , Sanka , S. , Rajarajan , M. and Nair , S.K. ( 2011 ), “ Capability-based cryptographic data access control in cloud computing ”, International Journal of Advanced Networking and Applications , Vol. 3 No. 3 , pp. 1152 - 1161 .
Hu , H. , Ahn , G.-J. and Jorgensen , J. ( 2012 ), “ Multiparty access control for online social networks: model and mechanisms ”, IEEE Transactions on Knowledge and Data Engineering , Vol. 25 No. 7 , pp. 1614 - 1627 .
Hu , V.C. , Ferraiolo , D. , Kuhn , R. , Schnitzer , A. , Sandlin , K. , Miller , R. and Scarfone , K. ( 2014 ), “ Guide to attribute based access control (ABAC) definition and considerations ”, NIST Special Publication , Vol. 800 , p. 162 , doi: 10.6028/NIST.SP.800-162 , available at: https://nvlpubs.nist.gov/nistpubs/specialpublications/NIST.SP.800-162.pdf
Hu , V.C. , Ferraiolo , D.F. , Chandramouli , R. and Kuhn , D.R. ( 2017b ), Attribute-Based Access Control , Artech House .
Hu , V.C. and Scarfone , K. ( 2012 ), Guidelines for Access Control System Evaluation Metrics , National Institute of Standards and Technology , Gaithersburg, MD , doi: 10.6028/NIST.IR.7874 .
Hu , V.C. , Kuhn , R. and Yaga , D. ( 2017a ), “ Verification and test methods for access control policies/models ”, NIST Special Publication , Vol. 800 , p. 192 .
Hu , V. ( 2016 ), “ Access control policy and implementation guides ”, available at: https://csrc.nist.gov/Projects/Access-Control-Policy-and-Implementation-Guides (accessed 14 April 2022 ).
IBM-Corporation ( 2015 ), “ Authentication versus access control ”, available at: www.ibm.com/docs/en/wca/3.5.0?topic=security-authentication-versus-access-control (accessed 14 April 2022 ).
Josang , A. ( 2017 ), “ A consistent definition of authorization ”, in Livraga , G. and Mitchell , C. (Eds), Security and Trust Management , Lecture Notes in Computer Science , Springer International Publishing , Cham , Vol. 10547 , pp. 134 - 144 , isbn: 978-3-319-68062-0 , doi: 10.1007/978-3-319-68063-7_9 .
Kalam , A.A.E. , Baida , R.E. , Balbiani , P. , Benferhat , S. , Cuppens , F. , Deswarte , Y. , Miége , A. , Saurel , C. and Trouessin , G. ( 2003 ), “ Organization based access control ”, in Proceedings POLICY 2003. IEEE 4th International Workshop on Policies for Distributed Systems and Networks , IEEE , pp. 120 - 131 .
Kane , K. and Browne , J.C. ( 2006 ), “ On classifying access control implementations for distributed systems ”, in Ferraiolo D. and Ray I. (Eds), Proceedings of the eleventh ACM symposium on Access control models and technologies , ACM Digital Library, ACM , New York, NY , p. 29 , isbn: 1595933530 , doi: 10.1145/1133058.1133064 .
Karatas , G. and Akbulut , A. ( 2018 ), “ Survey on access control mechanisms in cloud computing ”, Journal of Cyber Security and Mobility , doi: 10.13052/2245-1439.731 , ISSN: 2245-1439 .
Keefe , T.F. , Tsai , W.-T. and Srivastava , J. ( 1993 ), “ Database concurrency control in multilevel secure database management systems ”, IEEE Transactions on Knowledge and Data Engineering , Vol. 5 No. 6 , pp. 1039 - 1055 .
Kizza , J.M. ( 2020 ), “ Access control and authorization ”, Kizza , J.M. (Ed), Guide to Computer Network Security Texts in Computer Science , Springer International Publishing , Cham , pp. 187 - 206 , isbn: 978-3-030-38140-0 , doi: 10.1007/978-3-030-38141-7_9 .
Kriti , I.K. ( 2013 ), “ Database security and access control models: a brief overview ”, International Journal of Engineering Research and Technology (IJERT) , Vol. 2 No. 5 .
Kuhrmann , M. , Fernández , D.M. and Daneva , M. ( 2017 ), “ On the pragmatic design of literature studies in software engineering: an experiencebased guideline ”, Empirical Software Engineering , Vol. 22 No. 6 , pp. 2852 - 2891 .
Li , J. , Zhao , G. , Chen , X. , Xie , D. , Rong , C. , Li , W. , Tang , L. and Tang , Y. ( 2010 ), “ Fine-grained data access control systems with user accountability in cloud computing ”, In 2010 IEEE Second International Conference on Cloud Computing Technology and Science , IEEE , pp. 89 - 96 .
Majumder , A. , Namasudra , S. and Nath , S. ( 2014 ), “ Taxonomy and classification of access control models for cloud environments ”, in Zaigham , M. (Ed), Continued Rise of the Cloud, Computer Communications and Networks , Springer London , London , pp. 23 - 53 , isbn: 978-1-4471-6451-7 , doi: 10.1007/978-1-4471-6452-4_2 .
Matt , B. ( 2018 ), Computer Security: art and Science , Addison-Wesley Professional , isbn: 978-0-13-409714-5 .
Mell , P. and Grance , T , ( 2011 ), “ The NIST definition of cloud computing ”, National Institute of Science and Technology, Special Publication .
Molloy , I. , Dickens , L. , Morisset , C. , Cheng , P.-C. , Lobo , J. and Russo , A. ( 2012 ), “ Risk-based security decisions under uncertainty ”, in Proceedings of the 2nd ACM conference on Data and Application Security and Privacy , pp. 157 - 168 .
Ni , Q. , Bertino , E. and Lobo , J. ( 2010 ), “ Risk-based access control systems built on fuzzy inferences ”, in Proceedings of the 5th ACM Symposium on Information, Computer and Communications Security , pp. 250 - 260 .
OASIS ( 2013 ), “ Extensible access control markup language (XACML) version 3.0 ”, OASIS Open , available at: http://docs.oasis-open.org/xacml/3.0/xacml-3.0-core-spec-os-en.html (accessed 14 April 2022 ).
Park , J. and Sandhu , R. ( 2002 ), “ Originator control in usage control ”, in Proceedings of the 3rd International Workshop on Policies for Distributed Systems and Networks , IEEE , pp. 60 - 66 .
Petkovic , M. and Jonker , W. ( 2007 ), Security, Privacy, and Trust in Modern Data Management , Springer .
Qiu , J. , Tian , Z. , Du , C. , Zuo , Q. , Su , S. and Fang , B. ( 2020 ), “ A survey on access control in the age of internet of things ”, IEEE Internet of Things Journal , Vol. 7 No. 6 , pp. 4682 - 4696 , issn: 2327-4662 , doi: 10.1109/JIOT.2020.2969326 .
Rajbhandari , L. and Snekkenes , E.A. ( 2010 ), “ Using game theory to analyze risk to privacy: an initial insight ”, in IFIP PrimeLife International Summer School on Privacy and Identity Management for Life , Springer , pp. 41 - 51 .
Regateiro , D. , Pereira , Ó. and Aguiar , R. ( 2018 ), “ SeqBAC: a sequence based access control model ”, In Proceedings of the 30th International Conference on Software Engineering and Knowledge Engineering (SEKE 2018) , pp. 276 - 319 , doi: 10.18293/SEKE2018-099 .
Rjaibi , W. ( 2004 ), “ An introduction to multilevel secure relational database management systems ”, in Proceedings of the 2004 Conference of the Centre for Advanced Studies on Collaborative Research , pp. 232 - 241 .
Ross , R. , Pillitteri , V. , Graubart , R. , Bodeau , D. and Mcquaid , R. ( 2021 ), “ Developing cyber-resilient systems ”, Gaithersburg, MD , doi: 10.6028/NIST.SP.800-160v2r1 , available at https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-160v2r1.pdf (accessed 14 April 2022 ).
Sahafizadeh , E. and Parsa , S. ( 2010 ), “ Survey on access control models ”, in 2010 2nd International Conference on Future Computer and Communication , IEEE , Vol. 1 .
Samarati , P. and De Capitani di Vimercati , S. et al. ( 2001 ), “ Access control: policies, models, and mechanisms ”, in Goos , G. (Ed), Foundations of Security Analysis and Design , Lecture Notes in Computer Science , Springer Berlin Heidelberg , Berlin, Heidelberg , Vol. 2171 , pp. 137 - 196 , isbn: 978-3-540-42896-1 , doi: 10.1007/3-540-45608-2_3 .
Sandhu , R. , Coyne , E.J. , Youman , C.E. and Feinstein , H.L. ( 1996 ), “ Role-based access control models ”, Computer , Vol. 29 No. 2 , pp. 38 - 47 , doi: 10.1109/2.485845 .
Sandhu , R.S. and Samarati , P. ( 1994 ), “ Access control: principle and practice ”, IEEE Communications Magazine , Vol. 32 No. 9 , pp. 40 - 48 , ISSN: 0163-6804 , available at https://ieeexplore.ieee.org/document/312842 (accessed 14 April 2022 ).
Sandhu , R. , Bhamidipati , V. and Munawer , Q. ( 1999 ), “ The ARBAC97 model for role-based administration of roles ”, ACM Transactions on Information and System Security , Vol. 2 No. 1 , pp. 105 - 135 , doi: 10.1145/300830.300839 , issn: 1094-9224 .
Sifou , F. , Kartit , A. and Hammouch , A. ( 2017 ), “ Different access control mechanisms for data security in cloud computing ”, in Proceedings of the 2017 International Conference on Cloud and Big Data Computing , ACM , New York, NY , pp. 40 - 44 , isbn: 9781450353434 , doi: 10.1145/3141128.3141133 , available at: https://dl.acm.org/doi/pdf/10.1145/3141128.3141133 (accessed 14 April 2022 ).
Sun , L. and Wang , H. ( 2010 ), “ A purpose based usage access control model ”, International Journal of Computer and Information Engineering , Vol. 4 No. 1 , pp. 44 - 51 .
Tamizharasi , G.S. , Balamurugan , B. and Manjula , R. ( 2016 ), “ Attribute based encryption with fine-grained access provision in cloud computing ”, in Proceedings of the International Conference on Informatics and Analytics , pp. 1 - 4 .
Tapiador , A , Carrera , D. and Salvachúa , J. ( 2012 ), “ Tie-RBAC: an application of RBAC to social networks ”, CoRR abs/1205.5720 , arXiv: 1205.5720 , available at http://arxiv.org/abs/1205.5720
Wu , Y. , Suhendra , V. and Guo , H. ( 2012 ), “ A gateway-based access control scheme for collaborative clouds ”, in Proceedings of the 7th International Conference on Internet Monitoring and Protection , pp. 54 - 60 .
Xie , U. , Wen , H. , Wu , B. , Jiang , Y. and Meng , J. ( 2015 ), “ A modified hierarchical attribute-based encryption access control method for mobile cloud computing ”, IEEE Transactions on Cloud Computing , Vol. 7.2 , pp. 383 - 391 .
Xu , Y. , Zeng , Q. , Wang , G. , Zhang , C. , Ren , J. and Zhang , Y. ( 2018 ), “ A privacy-preserving attribute-based access control scheme ”, in International Conference on Security, Privacy and Anonymity in Computation, Communication and Storage , Springer , pp. 361 - 370 .
Zeng , W. , Yang , Y. and Luo , B. ( 2014 ), “ Content-based access control: use data content to assist access control for large-scale content-centric databases ”, in 2014 IEEE International Conference on Big Data , IEEE , pp. 701 - 710 .
Zhu , Y. , Hu , H. , Ahn , G.-J. , Huang , D. and Wang , S. ( 2012 ), “ Towards temporal access control in cloud computing ”, in 2012 Proceedings IEEE Infocom , IEEE , pp. 2576 - 2580 .
The research reported in this paper has been partly supported by the LIT Secure and Correct Systems Lab funded by the State of Upper Austria. The work was also funded within the FFG BRIDGE project KnoP-2D (grant no. 871299).
Related articles, all feedback is valuable.
Please share your general feedback
Contact Customer Support
Click through the PLOS taxonomy to find articles in your field.
For more information about PLOS Subject Areas, click here .
Loading metrics
Open Access
Peer-reviewed
Research Article
Contributed equally to this work with: Allison Langham-Putrow, Caitlin Bakker, Amy Riegelman
Roles Conceptualization, Investigation, Writing – original draft, Writing – review & editing
* E-mail: [email protected]
Affiliation University of Minnesota Libraries, University of Minnesota, Minneapolis, MN, United States of America
Roles Conceptualization, Data curation, Investigation, Writing – original draft, Writing – review & editing
Affiliation University of Minnesota Health Sciences Libraries, Minneapolis MN, United States of America
Over the last two decades, the existence of an open access citation advantage (OACA)—increased citation of articles made available open access (OA)—has been the topic of much discussion. While there has been substantial research to address this question, findings have been contradictory and inconclusive. We conducted a systematic review to compare studies of citations to OA and non-OA articles.
A systematic search of 17 databases attempted to capture all relevant studies authored since 2001. The protocol was registered in Open Science Framework. We included studies with a direct comparison between OA and non-OA items and reported article-level citation as an outcome. Both randomized and non-randomized studies were included. No limitations were placed on study design, language, or publication type.
A total of 5,744 items were retrieved. Ultimately, 134 items were identified for inclusion. 64 studies (47.8%) confirmed the existence of OACA, while 37 (27.6%) found that it did not exist, 32 (23.9%) found OACA only in subsets of their sample, and 1 study (0.8%) was inconclusive. Studies with a focus on multiple disciplines were significantly positively associated with finding that OACA exists in subsets, and are less associated with finding that OACA did not exist. In the critical appraisal of the included studies, 3 were found to have an overall low risk of bias. Of these, one found that an OACA existed, one found that it did not, and one found that an OACA occurred in subsets.
As seen through the large number of studies identified for this review, OACA is a topic of continuing interest. Quality and heterogeneity of the component studies pose challenges for generalization. The results suggest the need for reporting guidelines for bibliometrics studies.
Citation: Langham-Putrow A, Bakker C, Riegelman A (2021) Is the open access citation advantage real? A systematic review of the citation of open access and subscription-based articles. PLoS ONE 16(6): e0253129. https://doi.org/10.1371/journal.pone.0253129
Editor: Sergi Lozano, Universitat de Barcelona, SPAIN
Received: February 19, 2021; Accepted: June 1, 2021; Published: June 23, 2021
Copyright: © 2021 Langham-Putrow et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data are available from https://doi.org/10.13020/357e-ek33 .
Funding: The authors received no funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Scholarly publishing relies on the gift economy of academia in which authors provide content for free and editors and peer reviewers donate their time to review that content [ 1 , 2 ]. Prior to the development of the internet, the costs of printing, shipping, and other operations were offset by user and institutional subscriptions. As publishing became more commercialized in the second half of the 20th century, journal subscription prices began increasing, outpacing both inflation and university budgets [ 3 – 7 ]. Despite the shift from print to electronic, prices have only risen.
Various modes of open access (OA) publishing have emerged over the last 30 years to replace or augment subscription-based (“toll-access”) publishing. Green OA refers to materials that are made openly available through archiving in an OA repository or other legally permissible venue. Depending on journal policies and the decision-making of the depositor, a green OA version could be an early version of a manuscript or a copy-edited, post-peer review version. Historically, Gold OA was used to refer to “OA delivered by journals, regardless of the journal’s business model” [ 8 p. 52]. Gold OA can be further broken into gold journals (fully OA journals, which may or may not charge article processing charges (APCs)) and hybrid journals (which charge authors an APC to make their article open in a subscription journal).
Over the last 20 years, researchers and publishers have introduced a multitude of categories of OA beyond green and gold. These include "diamond" or "platinum" OA (i.e., OA in the place of publication but without an APC charged [ 9 , 10 ]), “bronze” OA (i.e., articles that publishers make available to readers for free, but do not have an OA license [ 11 ]), and "grey" OA (i.e., authors make their content available via upload to an academic social network or personal website, potentially without consideration of previously signed copyright transfer agreements). Others have used the term “black OA” to mean access facilitated by the posting of materials to pirate sites, such as SciHub [ 12 ]. It is important to note that this type of access does not necessarily involve the consent or participation of the author or publisher; as such, it is questionable whether this is truly a mode of OA. Because terminology has developed over time, we refer only to green OA versus gold OA, as these are the most broadly adopted categories of OA.
The potential for OA publication to increase citations was first articulated in an empirical study of computer science in 2001 [ 13 ]. Since 2001, there have been many studies both supporting and refuting the existence of an open access citation advantage (OACA). This nomenclature is consistently used in the literature, even in those studies reporting negative or null findings.
Although many relevant primary studies and some secondary studies exist on this topic, the influence of OA on citation remains unclear, particularly when considering the range of disciplines, data sources, publishing models, and other contextual factors. In a critical review, Davis and Walters [ 14 ] found a notable increase in the number of downloads for articles published OA but no clear evidence of OACA. Additionally, Davis and Walters found that studies finding citation advantages for OA failed to “adequately control for confounding variables” [ 14 p. 208]. In a narrative review, Turk [ 15 ] noted that OACA studies varied greatly in terms of data collection platforms used (e.g., Scopus, Google Scholar), differing publication types (e.g., journal articles, conference papers), and different disciplines analyzed, which potentially influenced the variety of findings.
In an update to the literature review by Davis and Walters [ 14 ], Lewis [ 16 ] concluded that only a “few of the authors…actually claim causation”, and instead pointed to the insight gained by examining correlation between OA and a citation advantage (CA). Lewis argued that more research is needed "to prove a causal relationship between OA and CA" [ 16 p. 59]. Like Turk [ 15 ] and others, Lewis noted that differences existed in studies targeting certain disciplines, and future research across all disciplines would "provide a wider array of evidence for the occurrence of field-specific OACA and therefore of a more widespread OACA" [ 16 p. 59].
A 2007 critical review by Craig et al. looked at three non-exclusive postulates when examining citation differences between both OA and non-OA articles: that the advantage is due to 1) OA status of the article; 2) selection bias (i.e., authors select their best works to share openly), and 3) early view effects (i.e., the extra time that an article is available) [ 17 ]. They pointed to studies in which controlling for one postulate (e.g., early view) revealed that citation counts can be explained by another postulate (e.g., selection bias) and advised that more rigorous methods were needed in future studies to look at causation [ 17 ].
Hua et al. [ 18 ] published a narrative review specific to OA concepts in dentistry in which they go beyond OACA and venture into other topics such as research waste of inaccessible research. Regarding OACA, they looked at nine studies across many disciplines and found variation of methods and materials, but ultimately the citation advantage ranged from -5% to 83% [ 18 ]. Turk [ 19 ] published an overview on OA, pertaining only to medical articles, looking at OA factors in which studies of possible OACA mostly found a citation advantage, but articles reporting randomized controlled trial (RCT) results revealed no OACA. Turk [ 19 ] indicated that other factors may influence a citation advantage for OA such as discipline, impact factor of journals studied, and the early view effect and selection bias Craig noted in 2007 [ 17 ].
Although these reviews provide valuable context, they were not as comprehensive and the methods were not transparently reported to the level of rigor needed to ensure reproducibility or replicability. The present study is unique in that there are no other existing systematic reviews or meta-analyses comparing citations of OA and non-OA articles. We address this need and critically appraise the existing studies to better understand the totality of the evidence while also identifying future areas of research. These findings could be of value to researchers when making evidence-based decisions about which mode of publishing—OA or otherwise—may most closely align with some of their objectives when choosing a publication venue.
In accordance with best practices for systematic reviews, a combination of controlled vocabulary and natural language searching was used to comprehensively capture relevant studies ( S1 Checklist ). The search included iterations of terms for OA and citation advantage. A primary database search strategy was developed in Ovid Medline and then translated to the following databases reflecting a range of disciplines: Library & Information Science Source (LISS), Library Information Science & Technology Abstracts (LISTA), ERIC, Academic Search Premier, and Business Source Premier via EBSCO; PubMed; Embase, CAB Abstracts and PsycINFO via Ovid; Scopus; Web of Science Core Collection; Compendex; Sociological Abstracts, EconLit, and Dissertations & Theses Global via ProQuest; and SHARE. A full search strategy of our primary database is available in S1 Appendix .
As indicated in our inclusion and exclusion criteria, this project targeted only studies published after 2001, which is the date of the earliest known article on citation advantage of OA [ 13 ]. By study, we refer to those items that were retrieved, screened, and included in this project. By articles, we refer to the subject of those studies: the OA and non-OA materials for which citation counts were being gathered. The searches were initially conducted in July 2019 and were rerun in November 2020. We did not include any filters to limit language, study design, or publication type. To ensure that no potentially relevant items were overlooked, hand searching of reference lists was undertaken. In cases where limited or unclear data were available, authors were contacted for clarification and to seek additional information. Any non-English language content was translated into English for screening and, if applicable, extraction and assessment. The protocol was registered in Open Science Framework (osf.io/p2a7q).
Two screeners independently reviewed every title and abstract, applying previously determined inclusion and exclusion criteria. In order to be included, studies needed to contain a direct comparison between a sample of OA articles and non-OA articles, with citation counts as an outcome, and have been published since January 2001. We excluded studies that did not report citation count data, such as narrative reviews, editorials, or opinion pieces; did not include citation counts as an outcome; and/or did not directly compare OA versus non-OA articles. For example, a study that reported on citation counts for a set of OA articles against a different set of OA articles would be excluded; a study that compared citations to gold OA articles with citations to subscription articles would be included even if the authors did not take into account the possible overlap of green OA articles within either set.
Screening was completed using Rayyan, a web application that facilitates independent screening [ 20 ]. Where there were discrepancies, the conflicts were resolved via discussion or by a third person tie-breaker when necessary. Full-text screening was also completed by two independent screeners, and reasons for exclusion were recorded and are reported in Fig 1 .
PRISMA flowchart showing number of studies at each step of the selection process.
https://doi.org/10.1371/journal.pone.0253129.g001
The data extraction form was developed and piloted by three researchers. Data elements extracted included information about the study, including the publication type of the study, as well as the type of metrics used to measure citations, mode of OA (green, gold, or a combination), and overall findings of the study. With respect to categorization of the mode of OA, we relied on the study authors’ reporting of OA status. As noted in the introduction, more nuanced definitions of OA exist. However, these broader categories are more commonly used and have been in effect for longer, thus making them more applicable for the purposes of this study. As hybrid and gold OA were sometimes conflated within the included studies, these categories are combined in this project. Similarly, some authors used “green OA” to describe whether a version of the article was findable through a search engine (i.e., available in an OA repository or academic social network but not through a pirate site).
We extracted data elements describing the metrics used and the findings of the study. These elements were categorized into larger groupings for further analysis. Metrics were grouped into mean, median, total, and other, representing the most common methods of measurement, while overall findings were grouped as OACA exists, OACA does not exist, or OACA exists in subsets of the data. We extracted the disciplinary subject of each study, and subsequently mapped these subjects to the six major subject codes outlined by the Organisation for Economic Cooperation and Development (OECD)’s Field of Science and Technology Classification [ 21 ]. Use of OECD overcomes challenges in using database or publisher-specific resources, such as taxonomies from Elsevier or Clarivate, while providing a framework applicable to a broad range of disciplines and document types. The disciplinary subject of each study was defined as the disciplinary subject of the articles included in the study, rather than the discipline of the journal that published the study. Where the subject matter reflected more than one classification, the item was marked as “multiple disciplines.” Data extraction was completed independently by two researchers, as was risk of bias assessment. The Evidence-based Librarianship (EBL) critical appraisal tool developed by Glynn [ 22 ] was used, as this tool was developed for library and information science literature and therefore would be the most broadly applicable to the literature included in this study. Where there were any discrepancies in the data extracted or the risk of bias assessment, these discrepancies were resolved through discussion or by a third party when necessary.
Following data extraction, heterogeneity of the studies was assessed using the I 2 statistic to determine if quantitative synthesis would be appropriate. I 2 was calculated using RevMan 5.3 [ 23 ]. An I 2 of 90% was found among studies with sufficient data for pooling. As such, the studies were found to be significantly heterogeneous and quantitative synthesis was deemed inappropriate. Although we were not able to pool the data in a meta-analysis, chi-squared tests were conducted to analyze the associations between findings, as categorical variables, and their associations with characteristics of studies. These tests were performed using R 3.6.0 [ 24 ].
5,697 results were retrieved through database searching, and 47 additional items were retrieved through handsearching. Once duplicates were removed, 4,019 items were subject to title-abstract screening. Of these, 3,677 were excluded, leaving 342 items for full-text assessment. 208 items were excluded at the full-text screening phase, resulting in 134 studies that were ultimately included in this analysis [ 11 , 25 – 157 ].
Of the 134 items, more fully described in Table 1 , 132 were non-randomized studies while 2 were randomized. The studies most frequently addressed multiple disciplines (n = 45, 33.6%), followed by Medical and Health Sciences (n = 36, 26.9%), Natural Sciences (n = 22, 16.4%), and Social Sciences (n = 21, 15.7%). Detailed information on included studies is available in S1 Dataset .
https://doi.org/10.1371/journal.pone.0253129.t001
The studies used a range of data sources. Most frequently, data were retrieved from Clarivate/Web of Science (n = 82, 61.2%), followed by Scopus (n = 38, 28.4%) and Google Scholar (n = 29, 21.6%), including Google Scholar data accessed through Publish or Perish [ 158 ]. 19 (14.2%) studies used other sources of data, and 24 (17.9%) studies used more than one source of data.
The majority of the studies (n = 64, 47.8%) found that there was OACA while 27.6% (n = 37) found that it does not exist. 23.9% (n = 32) of studies found that, while there was no overall OACA, there was an advantage in subsets, such as for specific journals, certain periods of time, or subdisciplines. One study (0.8%) was inconclusive in its findings. These studies based their conclusions on a range of measures.
The characteristics of the 134 studies and their associated findings are described in Table 2 .
https://doi.org/10.1371/journal.pone.0253129.t002
There was no statistically significant relationship between mode of OA and findings. Findings that OACA existed were most common regardless of mode of OA: 72% (18/25) of studies focused on green OA, 39.6% (21/53) of those focused on gold OA, 39.4% (13/33) of those focused on both green and gold OA, and 52.2% (12/23) of items that did not specify mode found that OACA existed. While 39.6% of studies focused on gold OA found OACA, gold studies account for 51.4% (19/37) of all studies that found no OACA existed.
We found a statistically significant relationship between discipline and OACA finding (X 2 (18,134) = 42.763,p < 0.001). Studies addressing multiple disciplines were positively associated with finding that OACA sometimes exists, and are less associated with finding that OACA did not exist. While not statistically significant associations, findings of the existence of OACA were more common in Social Sciences (66.7%, 14/21), Medical and Health Sciences (52.8%, 19/36), and Natural Sciences (45.5%, 10/22). In both Agricultural Sciences and studies that did not specify discipline, 25% of studies found that OACA existed, while OACA was not found in 75% of studies. Of the 32 studies that found OACA was present in subsets, 20 (62.5%) addressed multiple disciplines.
The EBL critical appraisal tool considers the possibility of bias in four domains: (1) population, (2) data collection, (3) study design, and (4) results [ 22 ]. Of the 134 studies assessed, 3 were found to have an overall low risk of bias. Of the three studies that had low overall risk of bias, one found that an OACA existed, one found that it did not, and one found that an OACA occurred in subsets.
All studies were found to have high risk of bias in the domain of population, 118 (88.1%) studies were found to have high risk of bias in the results domain, 107 (79.9%) were high risk of bias in data collection, and 91 (67.9%) had high risk of bias in study design. A summary of risk of bias assessment can be found in Fig 2 , while an itemized risk of bias assessment for every component study can be found in S1 Dataset . There were no statistically significant associations between overall risk of bias and findings.
Summary of overall risk of bias findings (low or high) and in the population, data collection, study design, and results domains.
https://doi.org/10.1371/journal.pone.0253129.g002
The majority of studies were published as journal articles (107/134), regardless of findings. Of those studies published as journal articles, the majority were published in subscription or hybrid journals (75/107). The remaining studies were made available through conference presentations (15/134), as working papers or preprints (8/134), or as theses or dissertations (4/134). There were no statistically significant relationships between publication type and findings of the studies.
The concept of OACA has been discussed since before the various declarations and definitions of OA [ 159 – 161 ]. In 2001, Lawrence reported that “free online availability substantially increases a paper’s impact.” [13 p. 521] As seen in our study, the topic has been the subject of numerous studies over the last 20 years.
Due to the heterogeneity of the sample of studies, a meta-analysis was not possible. We noted variations in the mode of OA, the types of publications, and the disciplines studied. There was also variation in the methods used and metrics reported. This suggests the need for reporting guidelines for bibliometric studies. At minimum, studies should report clearly on the mode of OA, the number of OA and non-OA articles, comparability of the data used, the time frame (citation window) and a justification for this choice, a common metric (e.g., mean and median citations per article), and confounding factors and how they are addressed.
With respect to the OA mode, studies should provide their definition of the mode of OA being considered. This may become even more necessary as researchers continue to create and implement new OA definitions. Additionally, if studying green OA, the study should assure that the articles in the non-OA population remain closed throughout the study and specify their exact working definition of “green” (e.g., whether articles found on academic social networks or author web pages are included). Similarly, studies exploring gold OA should consider whether articles in the OA sample are open throughout the entire citation window and the possibility that “non-OA” articles may have been openly available through other modes. The first consideration is complicated not only by potential delays in archiving an open copy, but also by new modes of OA. For example, with the introduction of bronze OA, publishers are free to move articles without an open license behind or in front of a paywall at any time and the status of an article at the time of data collection may differ from its previous status.
Although many studies reported metrics of mean citations per article and median citations per article, we found a wide variety of metrics studies reported. These included time-based metrics (e.g., citations per month since publication) and proportional metrics (e.g., difference in citations per paper, ratio of cited to non-cited articles). Other studies explored OACA by building linear regression models. The variations in methodology are not surprising, given the wide range of authors’ fields of research, including library and information studies, health sciences, computer science, and physics. OACA studies are published in bibliometric-focused journals, but also in discipline-specific journals. Disciplinary differences may explain some of the variation of methods employed. While a range of outcome measures can prove challenging, the consistent sharing of underlying data can help to overcome these barriers.
Very few (1.5%) of the studies in our sample were RCTs. There are notable challenges to conducting a randomized study on this topic. Beyond issues of funding, it may prove difficult to obtain consent from authors to randomize their articles to either condition. Instead, studies look retrospectively at populations of previously published articles. Nonrandomized studies face the challenge of ensuring that studies are truly comparable at baseline, allowing researchers to control for confounding factors including the potential "novelty" of the study, interest in the study, author reputation, and journal prestige [ 162 ]. Studies may also need to address the possibility of surreptitious cointerventions, such as toll access articles being made available through venues like ResearchGate, regardless of the journal’s OA policy. Although the prevalence of nonrandomized studies is understandable, these potential confounders pose challenges for identifying the implications of publishing decisions.
Out of the 134 studies included, 40 acknowledged the possibility of confounders, although not all 40 subsequently controlled for confounders in their analyses. Confounders noted included years since publication, Journal Impact Factor, number of authors, length of article, type of study (e.g., empirical or otherwise), prominence in search engine results, and the alphabetical position of the first author [ 77 , 91 , 136 , 140 , 142 , 148 , 149 ]. While examination of confounders is an important component of analysis, we found that this was inconsistently done and the confounders being considered varied significantly between studies.
We found high risk of bias in nearly 98% of the included studies ( Fig 2 ). High risk of bias/low validity in the population domain was often due to a poorly described sample or use of too narrow a sample to support the conclusions drawn. Few studies provided justification for their sample size. We saw high risk of bias/low validity in data collection, often due to lack of justification for why a particular time frame was used (i.e., length of the citation window). Citations accrue over time, at rates that can vary across disciplines. OACA studies should take into account the citation patterns for the discipline being studied. High risk of bias/low validity in study design was often due to poor reporting of outcomes in relation to data collection.
Because of the limitations of the quality of the studies in our review, it is not possible to draw definitive conclusions and recommendations for authors deciding whether to make their work OA. We also recognize that venue choice is complex, and rarely driven by a single factor. Authors may be required to make work public or OA by funding agency mandates, or they may wish to do so to reach certain audiences. These decisions may be complicated, and arrived at through collaboration and discussion, and may be influenced by external factors such as career stage, departmental and organizational requirements, and disciplinary norms. In a survey of over 2,100 researchers at R1 institutions, respondents named journal reputation and quality, alignment between the article and journal scope, and the journal’s readership as the most important factors when selecting a journal, while the OA status of the journal was rated to be the least important consideration in the journal selection process, with 12% of respondents considering it to be very important and 18% of respondents considering it to be important [ 163 ]. Although the OA status of the journal seems to be relatively unimportant to many researchers, the potential impact of OA on citations remains an ongoing discussion.
One reason there is so much interest in whether OACA exists is due to the emphasis placed on citation metrics in retention, promotion, and tenure (RPT) decisions. Aksnes et al. [ 164 ] provide an overview of the use of citation metrics to evaluate the quality of research and whether citations accurately reflect quality or impact. The Scholarly Communications Lab ( https://www.scholcommlab.ca/ ) has explored the use of citation metrics in RPT. Alperin et al. [ 165 ] analyzed 129 RPT documents from US and Canadian universities and found references to metrics in more than 70% of documents from research-intensive universities and nearly half of master’s colleges and universities.
However, citation metrics only measure the use of a study in the academic world. The goal of OA is to enable broader access to research; these uses may not be captured through citations. Scholars wish to publish in the venues that reach their audiences, for example, to reach practitioners who do not have institutional access to subscription resources. These uses may not result in subsequent citation in scholarly work, but are valuable nonetheless. Conversations around the need to and methods of acknowledging impact outside of an academic space have been ongoing, with organizations such as the European Commission noting that “[t]he exclusive use of bibliometric parameters as proxies for excellence in assessment by most funding agencies and universities/research organisations does not facilitate Open Science” [ 166 p. 8].
There are limitations to our study. Most notably, we were unable to conduct a quantitative meta-analysis due to the heterogeneity of and the high risk of bias in our pool of studies. Our findings of high risk of bias for nearly all of the studies may be reflective of the lack of reporting guidelines. Reporting guidelines are “[a] checklist, flow diagram, or structured text to guide authors in reporting a specific type of research, developed using explicit methodology,” providing a list of the minimum information to be shared in each section of a paper [ 167 ]. Establishing reporting guidelines for bibliometrics studies and meta-research would be valuable in improving the completeness, clarity, and quality of studies in this area. Scholars would continue to have freedom regarding study design but have guidance on what details would need to be transparently reported. We also note that the EBL critical appraisal tool was designed for library and information research, but not specifically for bibliometric studies [ 22 ]. [To the best of our knowledge, there is no risk of bias assessment specifically designed for bibliometric studies, and the unique features of those studies. Development of such a tool, or modification of an existing tool for this purpose, could be of value. Although we did conduct searches across 17 databases, it is possible that relevant resources were not included. Non-English language content and publishers from outside of North America, the UK, and western Europe are underrepresented in scholarly databases, which may have resulted in potential omission in our study [ 105 , 168 ].
The understanding and use of alternative metrics ("altmetrics”) has increased over the last 10 years [ 169 ]. Altmetrics measure the attention a work receives through metrics such as number of downloads, shares, or tweets on Twitter. These metrics have the potential to reflect use of research that would not be captured through scholarly citations, such as articles that are downloaded by healthcare providers and used in clinical care. Some of the studies in our sample addressed altmetrics (alternative metrics, such as number of downloads or views of an article). Studies included in our analysis occasionally considered altmetrics along with traditional citations (15%). An “open access altmetric advantage” could be evaluated through a review like this one.
As seen through the large number of studies identified for this review, OACA is a topic of high and continuing interest. In a now discontinued service, SPARC Europe previously tracked findings of studies measuring OACA [ 170 ]. Although this was a worthy effort, data collection ceased in 2015, the methods of identifying the included studies are not transparent or reproducible, and no risk of bias assessments were conducted. Scholars and institutions continue to wrestle with difficult decisions regarding OA publishing amidst a variety of funding arrangements. More rigorous and robustly reported primary research and follow-up syntheses are needed to equip stakeholders with evidence to make informed choices.
OACA studies could be continuously tracked and reviewed qualitatively, if not quantitatively, in the mode of an emerging method, Living Systematic Reviews (LSRs). The purpose of a LSR is to incorporate new studies as they are published [ 171 ]. Further, a LSR could be used to regularly communicate the review status updates with stakeholders as the new evidence is incorporated. Whether or not there is an OACA is a topic of continuing interest and a LSR could support rapidly incorporating new evidence.
S1 checklist. prisma checklist..
This file the PRISMA 2009 checklist with page numbers.
https://doi.org/10.1371/journal.pone.0253129.s001
This file contains the primary database search strategy that was developed in Ovid Medline.
https://doi.org/10.1371/journal.pone.0253129.s002
This table contains data for all of the included studies.
https://doi.org/10.1371/journal.pone.0253129.s003
We would like to thank Jonathan Koffel and Emma Molls for their critical reading of the manuscript.
Run a free plagiarism check in 10 minutes, generate accurate citations for free.
Methodology
Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.
A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.
They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”
In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.
What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.
A review is an overview of the research that’s already been completed on a topic.
What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:
Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.
Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.
Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.
Discover proofreading & editing
Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.
A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .
A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.
Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.
Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.
However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.
Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.
Professional editors proofread and edit your paper by focusing on:
See an example
A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.
To conduct a systematic review, you’ll need the following:
A systematic review has many pros .
Systematic reviews also have a few cons .
The 7 steps for conducting a systematic review are explained with an example.
Formulating the research question is probably the most important step of a systematic review. A clear research question will:
A good research question for a systematic review has four components, which you can remember with the acronym PICO :
You can rearrange these four components to write your research question:
Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .
Their research question was:
A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.
Your protocol should include the following components:
If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.
It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .
Searching for relevant studies is the most time-consuming step of a systematic review.
To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:
At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .
Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.
To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.
If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.
You should apply the selection criteria in two phases:
It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .
Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.
When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.
Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:
You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .
Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.
They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.
Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:
Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.
Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.
The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.
Your article should include the following sections:
To verify that your report includes everything it needs, you can use the PRISMA checklist .
Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.
In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.
If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.
Research bias
A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .
It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.
A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other academic texts , with an introduction , a main body, and a conclusion .
An annotated bibliography is a list of source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a paper .
A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.
If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.
Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved July 27, 2024, from https://www.scribbr.com/methodology/systematic-review/
Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, get unlimited documents corrected.
✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Email citation, add to collections.
Your saved search, create a file for external citation management software, your rss feed.
Affiliations.
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.
Keywords: evidence; guide; meta-analysis; meta-synthesis; narrative; systematic review; theory.
PubMed Disclaimer
Full text sources.
NCBI Literature Resources
MeSH PMC Bookshelf Disclaimer
The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.
Property | Value |
---|---|
Status | |
Version | |
Ad File | |
Disable Ads Flag | |
Environment | |
Moat Init | |
Moat Ready | |
Contextual Ready | |
Contextual URL | |
Contextual Initial Segments | |
Contextual Used Segments | |
AdUnit | |
SubAdUnit | |
Custom Targeting | |
Ad Events | |
Invalid Ad Sizes |
Access provided by
If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Measurable outcome/analysis, conclusions, purchase one-time access:, sneb member login, article info, identification.
DOI: https://doi.org/10.1016/j.jneb.2024.05.149
Related articles.
The content on this site is intended for healthcare professionals.
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .
Kat kolaski.
1 Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, NC USA
2 Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, NY USA
3 Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, CA USA
Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.
A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.
Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.
The online version contains supplementary material available at 10.1186/s13643-023-02255-9.
Evidence syntheses are commonly regarded as the foundation of evidence-based medicine (EBM). They are widely accredited for providing reliable evidence and, as such, they have significantly influenced medical research and clinical practice. Despite their uptake throughout health care and ubiquity in contemporary medical literature, some important aspects of evidence syntheses are generally overlooked or not well recognized. Evidence syntheses are mostly retrospective exercises, they often depend on weak or irreparably flawed data, and they may use tools that have acknowledged or yet unrecognized limitations. They are complicated and time-consuming undertakings prone to bias and errors. Production of a good evidence synthesis requires careful preparation and high levels of organization in order to limit potential pitfalls [ 1 ]. Many authors do not recognize the complexity of such an endeavor and the many methodological challenges they may encounter. Failure to do so is likely to result in research and resource waste.
Given their potential impact on people’s lives, it is crucial for evidence syntheses to correctly report on the current knowledge base. In order to be perceived as trustworthy, reliable demonstration of the accuracy of evidence syntheses is equally imperative [ 2 ]. Concerns about the trustworthiness of evidence syntheses are not recent developments. From the early years when EBM first began to gain traction until recent times when thousands of systematic reviews are published monthly [ 3 ] the rigor of evidence syntheses has always varied. Many systematic reviews and meta-analyses had obvious deficiencies because original methods and processes had gaps, lacked precision, and/or were not widely known. The situation has improved with empirical research concerning which methods to use and standardization of appraisal tools. However, given the geometrical increase in the number of evidence syntheses being published, a relatively larger pool of unreliable evidence syntheses is being published today.
Publication of methodological studies that critically appraise the methods used in evidence syntheses is increasing at a fast pace. This reflects the availability of tools specifically developed for this purpose [ 4 – 6 ]. Yet many clinical specialties report that alarming numbers of evidence syntheses fail on these assessments. The syntheses identified report on a broad range of common conditions including, but not limited to, cancer, [ 7 ] chronic obstructive pulmonary disease, [ 8 ] osteoporosis, [ 9 ] stroke, [ 10 ] cerebral palsy, [ 11 ] chronic low back pain, [ 12 ] refractive error, [ 13 ] major depression, [ 14 ] pain, [ 15 ] and obesity [ 16 , 17 ]. The situation is even more concerning with regard to evidence syntheses included in clinical practice guidelines (CPGs) [ 18 – 20 ]. Astonishingly, in a sample of CPGs published in 2017–18, more than half did not apply even basic systematic methods in the evidence syntheses used to inform their recommendations [ 21 ].
These reports, while not widely acknowledged, suggest there are pervasive problems not limited to evidence syntheses that evaluate specific kinds of interventions or include primary research of a particular study design (eg, randomized versus non-randomized) [ 22 ]. Similar concerns about the reliability of evidence syntheses have been expressed by proponents of EBM in highly circulated medical journals [ 23 – 26 ]. These publications have also raised awareness about redundancy, inadequate input of statistical expertise, and deficient reporting. These issues plague primary research as well; however, there is heightened concern for the impact of these deficiencies given the critical role of evidence syntheses in policy and clinical decision-making.
Several international consortiums of EBM experts and national health care organizations currently provide detailed guidance (Table (Table1). 1 ). They draw criteria from the reporting and methodological standards of currently recommended appraisal tools, and regularly review and update their methods to reflect new information and changing needs. In addition, they endorse the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system for rating the overall quality of a body of evidence [ 27 ]. These groups typically certify or commission systematic reviews that are published in exclusive databases (eg, Cochrane, JBI) or are used to develop government or agency sponsored guidelines or health technology assessments (eg, National Institute for Health and Care Excellence [NICE], Scottish Intercollegiate Guidelines Network [SIGN], Agency for Healthcare Research and Quality [AHRQ]). They offer developers of evidence syntheses various levels of methodological advice, technical and administrative support, and editorial assistance. Use of specific protocols and checklists are required for development teams within these groups, but their online methodological resources are accessible to any potential author.
Guidance for development of evidence syntheses
Cochrane (formerly Cochrane Collaboration) | |
JBI (formerly Joanna Briggs Institute) | |
National Institute for Health and Care Excellence (NICE)—United Kingdom | |
Scottish Intercollegiate Guidelines Network (SIGN) —Scotland | |
Agency for Healthcare Research and Quality (AHRQ)—United States |
Notably, Cochrane is the largest single producer of evidence syntheses in biomedical research; however, these only account for 15% of the total [ 28 ]. The World Health Organization requires Cochrane standards be used to develop evidence syntheses that inform their CPGs [ 29 ]. Authors investigating questions of intervention effectiveness in syntheses developed for Cochrane follow the Methodological Expectations of Cochrane Intervention Reviews [ 30 ] and undergo multi-tiered peer review [ 31 , 32 ]. Several empirical evaluations have shown that Cochrane systematic reviews are of higher methodological quality compared with non-Cochrane reviews [ 4 , 7 , 9 , 11 , 14 , 32 – 35 ]. However, some of these assessments have biases: they may be conducted by Cochrane-affiliated authors, and they sometimes use scales and tools developed and used in the Cochrane environment and by its partners. In addition, evidence syntheses published in the Cochrane database are not subject to space or word restrictions, while non-Cochrane syntheses are often limited. As a result, information that may be relevant to the critical appraisal of non-Cochrane reviews is often removed or is relegated to online-only supplements that may not be readily or fully accessible [ 28 ].
Many authors are familiar with the evidence syntheses produced by the leading EBM organizations but can be intimidated by the time and effort necessary to apply their standards. Instead of following their guidance, authors may employ methods that are discouraged or outdated 28]. Suboptimal methods described in in the literature may then be taken up by others. For example, the Newcastle–Ottawa Scale (NOS) is a commonly used tool for appraising non-randomized studies [ 36 ]. Many authors justify their selection of this tool with reference to a publication that describes the unreliability of the NOS and recommends against its use [ 37 ]. Obviously, the authors who cite this report for that purpose have not read it. Authors and peer reviewers have a responsibility to use reliable and accurate methods and not copycat previous citations or substandard work [ 38 , 39 ]. Similar cautions may potentially extend to automation tools. These have concentrated on evidence searching [ 40 ] and selection given how demanding it is for humans to maintain truly up-to-date evidence [ 2 , 41 ]. Cochrane has deployed machine learning to identify randomized controlled trials (RCTs) and studies related to COVID-19, [ 2 , 42 ] but such tools are not yet commonly used [ 43 ]. The routine integration of automation tools in the development of future evidence syntheses should not displace the interpretive part of the process.
Editorials about unreliable or misleading systematic reviews highlight several of the intertwining factors that may contribute to continued publication of unreliable evidence syntheses: shortcomings and inconsistencies of the peer review process, lack of endorsement of current standards on the part of journal editors, the incentive structure of academia, industry influences, publication bias, and the lure of “predatory” journals [ 44 – 48 ]. At this juncture, clarification of the extent to which each of these factors contribute remains speculative, but their impact is likely to be synergistic.
Over time, the generalized acceptance of the conclusions of systematic reviews as incontrovertible has affected trends in the dissemination and uptake of evidence. Reporting of the results of evidence syntheses and recommendations of CPGs has shifted beyond medical journals to press releases and news headlines and, more recently, to the realm of social media and influencers. The lay public and policy makers may depend on these outlets for interpreting evidence syntheses and CPGs. Unfortunately, communication to the general public often reflects intentional or non-intentional misrepresentation or “spin” of the research findings [ 49 – 52 ] News and social media outlets also tend to reduce conclusions on a body of evidence and recommendations for treatment to binary choices (eg, “do it” versus “don’t do it”) that may be assigned an actionable symbol (eg, red/green traffic lights, smiley/frowning face emoji).
Many authors and peer reviewers are volunteer health care professionals or trainees who lack formal training in evidence synthesis [ 46 , 53 ]. Informing them about research methodology could increase the likelihood they will apply rigorous methods [ 25 , 33 , 45 ]. We tackle this challenge, from both a theoretical and a practical perspective, by offering guidance applicable to any specialty. It is based on recent methodological research that is extensively referenced to promote self-study. However, the information presented is not intended to be substitute for committed training in evidence synthesis methodology; instead, we hope to inspire our target audience to seek such training. We also hope to inform a broader audience of clinicians and guideline developers influenced by evidence syntheses. Notably, these communities often include the same members who serve in different capacities.
In the following sections, we highlight methodological concepts and practices that may be unfamiliar, problematic, confusing, or controversial. In Part 2, we consider various types of evidence syntheses and the types of research evidence summarized by them. In Part 3, we examine some widely used (and misused) tools for the critical appraisal of systematic reviews and reporting guidelines for evidence syntheses. In Part 4, we discuss how to meet methodological conduct standards applicable to key components of systematic reviews. In Part 5, we describe the merits and caveats of rating the overall certainty of a body of evidence. Finally, in Part 6, we summarize suggested terminology, methods, and tools for development and evaluation of evidence syntheses that reflect current best practices.
A good foundation for the development of evidence syntheses requires an appreciation of their various methodologies and the ability to correctly identify the types of research potentially available for inclusion in the synthesis.
Systematic reviews have historically focused on the benefits and harms of interventions; over time, various types of systematic reviews have emerged to address the diverse information needs of clinicians, patients, and policy makers [ 54 ] Systematic reviews with traditional components have become defined by the different topics they assess (Table 2.1 ). In addition, other distinctive types of evidence syntheses have evolved, including overviews or umbrella reviews, scoping reviews, rapid reviews, and living reviews. The popularity of these has been increasing in recent years [ 55 – 58 ]. A summary of the development, methods, available guidance, and indications for these unique types of evidence syntheses is available in Additional File 2 A.
Types of traditional systematic reviews
Review type | Topic assessed | Elements of research question (mnemonic) |
---|---|---|
Intervention [ , ] | Benefits and harms of interventions used in healthcare. | opulation, ntervention, omparator, utcome ( ) |
Diagnostic test accuracy [ ] | How well a diagnostic test performs in diagnosing and detecting a particular disease. | opulation, ndex test(s), and arget condition ( ) |
Qualitative | ||
Cochrane [ ] | Questions are designed to improve understanding of intervention complexity, contextual variations, implementation, and stakeholder preferences and experiences. | etting, erspective, ntervention or Phenomenon of nterest, omparison, valuation ( ) ample, henomenon of nterest, esign, valuation, esearch type ( ) spective, etting, henomena of interest/Problem, nvironment, omparison (optional), me/timing, indings ( ) |
JBI [ ] | Questions inform meaningfulness and appropriateness of care and the impact of illness through documentation of stakeholder experiences, preferences, and priorities. | opulation, the Phenomena of nterest, and the ntext |
Prognostic [ ] | Probable course or future outcome(s) of people with a health problem. | opulation, ntervention (model), omparator, utcomes, iming, etting ( ) |
Etiology and risk [ ] | The relationship (association) between certain factors (e.g., genetic, environmental) and the development of a disease or condition or other health outcome. | opulation or groups at risk, xposure(s), associated utcome(s) (disease, symptom, or health condition of interest), the context/location or the time period and the length of time when relevant ( ) |
Measurement properties [ , ] | What is the most suitable instrument to measure a construct of interest in a specific study population? | opulation, nstrument, onstruct, utcomes ( ) |
Prevalence and incidence [ ] | The frequency, distribution and determinants of specific factors, health states or conditions in a defined population: eg, how common is a particular disease or condition in a specific group of individuals? | Factor, disease, symptom or health ndition of interest, the epidemiological indicator used to measure its frequency (prevalence, incidence), the ulation or groups at risk as well as the ntext/location and time period where relevant ( ) |
Both Cochrane [ 30 , 59 ] and JBI [ 60 ] provide methodologies for many types of evidence syntheses; they describe these with different terminology, but there is obvious overlap (Table 2.2 ). The majority of evidence syntheses published by Cochrane (96%) and JBI (62%) are categorized as intervention reviews. This reflects the earlier development and dissemination of their intervention review methodologies; these remain well-established [ 30 , 59 , 61 ] as both organizations continue to focus on topics related to treatment efficacy and harms. In contrast, intervention reviews represent only about half of the total published in the general medical literature, and several non-intervention review types contribute to a significant proportion of the other half.
Evidence syntheses published by Cochrane and JBI
Intervention | 8572 | 96.3 | Effectiveness | 435 | 61.5 |
Diagnostic | 176 | 1.9 | Diagnostic Test Accuracy | 9 | 1.3 |
Overview | 64 | 0.7 | Umbrella | 4 | 0.6 |
Methodology | 41 | 0.45 | Mixed Methods | 2 | 0.3 |
Qualitative | 17 | 0.19 | Qualitative | 159 | 22.5 |
Prognostic | 11 | 0.12 | Prevalence and Incidence | 6 | 0.8 |
Rapid | 11 | 0.12 | Etiology and Risk | 7 | 1.0 |
Prototype | 8 | 0.08 | Measurement Properties | 3 | 0.4 |
Economic | 6 | 0.6 | |||
Text and Opinion | 1 | 0.14 | |||
Scoping | 43 | 6.0 | |||
Comprehensive | 32 | 4.5 | |||
Total = 8900 | Total = 707 |
a Data from https://www.cochranelibrary.com/cdsr/reviews . Accessed 17 Sep 2022
b Data obtained via personal email communication on 18 Sep 2022 with Emilie Francis, editorial assistant, JBI Evidence Synthesis
c Includes the following categories: prevalence, scoping, mixed methods, and realist reviews
d This methodology is not supported in the current version of the JBI Manual for Evidence Synthesis
There is consensus on the importance of using multiple study designs in evidence syntheses; at the same time, there is a lack of agreement on methods to identify included study designs. Authors of evidence syntheses may use various taxonomies and associated algorithms to guide selection and/or classification of study designs. These tools differentiate categories of research and apply labels to individual study designs (eg, RCT, cross-sectional). A familiar example is the Design Tree endorsed by the Centre for Evidence-Based Medicine [ 70 ]. Such tools may not be helpful to authors of evidence syntheses for multiple reasons.
Suboptimal levels of agreement and accuracy even among trained methodologists reflect challenges with the application of such tools [ 71 , 72 ]. Problematic distinctions or decision points (eg, experimental or observational, controlled or uncontrolled, prospective or retrospective) and design labels (eg, cohort, case control, uncontrolled trial) have been reported [ 71 ]. The variable application of ambiguous study design labels to non-randomized studies is common, making them especially prone to misclassification [ 73 ]. In addition, study labels do not denote the unique design features that make different types of non-randomized studies susceptible to different biases, including those related to how the data are obtained (eg, clinical trials, disease registries, wearable devices). Given this limitation, it is important to be aware that design labels preclude the accurate assignment of non-randomized studies to a “level of evidence” in traditional hierarchies [ 74 ].
These concerns suggest that available tools and nomenclature used to distinguish types of research evidence may not uniformly apply to biomedical research and non-health fields that utilize evidence syntheses (eg, education, economics) [ 75 , 76 ]. Moreover, primary research reports often do not describe study design or do so incompletely or inaccurately; thus, indexing in PubMed and other databases does not address the potential for misclassification [ 77 ]. Yet proper identification of research evidence has implications for several key components of evidence syntheses. For example, search strategies limited by index terms using design labels or study selection based on labels applied by the authors of primary studies may cause inconsistent or unjustified study inclusions and/or exclusions [ 77 ]. In addition, because risk of bias (RoB) tools consider attributes specific to certain types of studies and study design features, results of these assessments may be invalidated if an inappropriate tool is used. Appropriate classification of studies is also relevant for the selection of a suitable method of synthesis and interpretation of those results.
An alternative to these tools and nomenclature involves application of a few fundamental distinctions that encompass a wide range of research designs and contexts. While these distinctions are not novel, we integrate them into a practical scheme (see Fig. Fig.1) 1 ) designed to guide authors of evidence syntheses in the basic identification of research evidence. The initial distinction is between primary and secondary studies. Primary studies are then further distinguished by: 1) the type of data reported (qualitative or quantitative); and 2) two defining design features (group or single-case and randomized or non-randomized). The different types of studies and study designs represented in the scheme are described in detail in Additional File 2 B. It is important to conceptualize their methods as complementary as opposed to contrasting or hierarchical [ 78 ]; each offers advantages and disadvantages that determine their appropriateness for answering different kinds of research questions in an evidence synthesis.
Distinguishing types of research evidence
Application of these basic distinctions may avoid some of the potential difficulties associated with study design labels and taxonomies. Nevertheless, debatable methodological issues are raised when certain types of research identified in this scheme are included in an evidence synthesis. We briefly highlight those associated with inclusion of non-randomized studies, case reports and series, and a combination of primary and secondary studies.
When investigating an intervention’s effectiveness, it is important for authors to recognize the uncertainty of observed effects reported by studies with high RoB. Results of statistical analyses that include such studies need to be interpreted with caution in order to avoid misleading conclusions [ 74 ]. Review authors may consider excluding randomized studies with high RoB from meta-analyses. Non-randomized studies of intervention (NRSI) are affected by a greater potential range of biases and thus vary more than RCTs in their ability to estimate a causal effect [ 79 ]. If data from NRSI are synthesized in meta-analyses, it is helpful to separately report their summary estimates [ 6 , 74 ].
Nonetheless, certain design features of NRSI (eg, which parts of the study were prospectively designed) may help to distinguish stronger from weaker ones. Cochrane recommends that authors of a review including NRSI focus on relevant study design features when determining eligibility criteria instead of relying on non-informative study design labels [ 79 , 80 ] This process is facilitated by a study design feature checklist; guidance on using the checklist is included with developers’ description of the tool [ 73 , 74 ]. Authors collect information about these design features during data extraction and then consider it when making final study selection decisions and when performing RoB assessments of the included NRSI.
Correctly identified case reports and case series can contribute evidence not well captured by other designs [ 81 ]; in addition, some topics may be limited to a body of evidence that consists primarily of uncontrolled clinical observations. Murad and colleagues offer a framework for how to include case reports and series in an evidence synthesis [ 82 ]. Distinguishing between cohort studies and case series in these syntheses is important, especially for those that rely on evidence from NRSI. Additional data obtained from studies misclassified as case series can potentially increase the confidence in effect estimates. Mathes and Pieper provide authors of evidence syntheses with specific guidance on distinguishing between cohort studies and case series, but emphasize the increased workload involved [ 77 ].
Synthesis of combined evidence from primary and secondary studies may provide a broad perspective on the entirety of available literature on a topic. This is, in fact, the recommended strategy for scoping reviews that may include a variety of sources of evidence (eg, CPGs, popular media). However, except for scoping reviews, the synthesis of data from primary and secondary studies is discouraged unless there are strong reasons to justify doing so.
Combining primary and secondary sources of evidence is challenging for authors of other types of evidence syntheses for several reasons [ 83 ]. Assessments of RoB for primary and secondary studies are derived from conceptually different tools, thus obfuscating the ability to make an overall RoB assessment of a combination of these study types. In addition, authors who include primary and secondary studies must devise non-standardized methods for synthesis. Note this contrasts with well-established methods available for updating existing evidence syntheses with additional data from new primary studies [ 84 – 86 ]. However, a new review that synthesizes data from primary and secondary studies raises questions of validity and may unintentionally support a biased conclusion because no existing methodological guidance is currently available [ 87 ].
We suggest that journal editors require authors to identify which type of evidence synthesis they are submitting and reference the specific methodology used for its development. This will clarify the research question and methods for peer reviewers and potentially simplify the editorial process. Editors should announce this practice and include it in the instructions to authors. To decrease bias and apply correct methods, authors must also accurately identify the types of research evidence included in their syntheses.
The need to develop criteria to assess the rigor of systematic reviews was recognized soon after the EBM movement began to gain international traction [ 88 , 89 ]. Systematic reviews rapidly became popular, but many were very poorly conceived, conducted, and reported. These problems remain highly prevalent [ 23 ] despite development of guidelines and tools to standardize and improve the performance and reporting of evidence syntheses [ 22 , 28 ]. Table 3.1 provides some historical perspective on the evolution of tools developed specifically for the evaluation of systematic reviews, with or without meta-analysis.
Tools specifying standards for systematic reviews with and without meta-analysis
Quality of Reporting of Meta-analyses (QUOROM) Statement | Moher 1999 [ ] |
Meta-analyses Of Observational Studies in Epidemiology (MOOSE) | Stroup 2000 [ ] |
Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) | Moher 2009 [ ] |
PRISMA 2020 | Page 2021 [ ] |
Overview Quality Assessment Questionnaire (OQAQ) | Oxman and Guyatt 1991 [ ] |
Systematic Review Critical Appraisal Sheet | Centre for Evidence-based Medicine 2005 [ ] |
A Measurement Tool to Assess Systematic Reviews (AMSTAR) | Shea 2007 [ ] |
AMSTAR-2 | Shea 2017 [ ] |
Risk of Bias in Systematic Reviews (ROBIS) | Whiting 2016 [ ] |
a Currently recommended
b Validated tool for systematic reviews of interventions developed for use by authors of overviews or umbrella reviews
These tools are often interchangeably invoked when referring to the “quality” of an evidence synthesis. However, quality is a vague term that is frequently misused and misunderstood; more precisely, these tools specify different standards for evidence syntheses. Methodological standards address how well a systematic review was designed and performed [ 5 ]. RoB assessments refer to systematic flaws or limitations in the design, conduct, or analysis of research that distort the findings of the review [ 4 ]. Reporting standards help systematic review authors describe the methodology they used and the results of their synthesis in sufficient detail [ 92 ]. It is essential to distinguish between these evaluations: a systematic review may be biased, it may fail to report sufficient information on essential features, or it may exhibit both problems; a thoroughly reported systematic evidence synthesis review may still be biased and flawed while an otherwise unbiased one may suffer from deficient documentation.
We direct attention to the currently recommended tools listed in Table 3.1 but concentrate on AMSTAR-2 (update of AMSTAR [A Measurement Tool to Assess Systematic Reviews]) and ROBIS (Risk of Bias in Systematic Reviews), which evaluate methodological quality and RoB, respectively. For comparison and completeness, we include PRISMA 2020 (update of the 2009 Preferred Reporting Items for Systematic Reviews of Meta-Analyses statement), which offers guidance on reporting standards. The exclusive focus on these three tools is by design; it addresses concerns related to the considerable variability in tools used for the evaluation of systematic reviews [ 28 , 88 , 96 , 97 ]. We highlight the underlying constructs these tools were designed to assess, then describe their components and applications. Their known (or potential) uptake and impact and limitations are also discussed.
Development.
AMSTAR [ 5 ] was in use for a decade prior to the 2017 publication of AMSTAR-2; both provide a broad evaluation of methodological quality of intervention systematic reviews, including flaws arising through poor conduct of the review [ 6 ]. ROBIS, published in 2016, was developed to specifically assess RoB introduced by the conduct of the review; it is applicable to systematic reviews of interventions and several other types of reviews [ 4 ]. Both tools reflect a shift to a domain-based approach as opposed to generic quality checklists. There are a few items unique to each tool; however, similarities between items have been demonstrated [ 98 , 99 ]. AMSTAR-2 and ROBIS are recommended for use by: 1) authors of overviews or umbrella reviews and CPGs to evaluate systematic reviews considered as evidence; 2) authors of methodological research studies to appraise included systematic reviews; and 3) peer reviewers for appraisal of submitted systematic review manuscripts. For authors, these tools may function as teaching aids and inform conduct of their review during its development.
Systematic reviews that include randomized and/or non-randomized studies as evidence can be appraised with AMSTAR-2 and ROBIS. Other characteristics of AMSTAR-2 and ROBIS are summarized in Table 3.2 . Both tools define categories for an overall rating; however, neither tool is intended to generate a total score by simply calculating the number of responses satisfying criteria for individual items [ 4 , 6 ]. AMSTAR-2 focuses on the rigor of a review’s methods irrespective of the specific subject matter. ROBIS places emphasis on a review’s results section— this suggests it may be optimally applied by appraisers with some knowledge of the review’s topic as they may be better equipped to determine if certain procedures (or lack thereof) would impact the validity of a review’s findings [ 98 , 100 ]. Reliability studies show AMSTAR-2 overall confidence ratings strongly correlate with the overall RoB ratings in ROBIS [ 100 , 101 ].
Comparison of AMSTAR-2 and ROBIS
Characteristic | ||
---|---|---|
Extensive | Extensive | |
Intervention | Intervention, diagnostic, etiology, prognostic | |
7 critical, 9 non-critical | 4 | |
Total number | 16 | 29 |
Response options | Items # 1, 3, 5, 6, 10, 13, 14, 16: rated or Items # 2, 4, 7, 8, 9 : rated or Items # 11 , 12, 15: rated or | 24 assessment items: rated 5 items regarding level of concern: rated |
Construct | Confidence based on weaknesses in critical domains | Level of concern for risk of bias |
Categories | High, moderate, low, critically low | Low, high, unclear |
a ROBIS includes an optional first phase to assess the applicability of the review to the research question of interest. The tool may be applicable to other review types in addition to the four specified, although modification of this initial phase will be needed (Personal Communication via email, Penny Whiting, 28 Jan 2022)
b AMSTAR-2 item #9 and #11 require separate responses for RCTs and NRSI
Interrater reliability has been shown to be acceptable for AMSTAR-2 [ 6 , 11 , 102 ] and ROBIS [ 4 , 98 , 103 ] but neither tool has been shown to be superior in this regard [ 100 , 101 , 104 , 105 ]. Overall, variability in reliability for both tools has been reported across items, between pairs of raters, and between centers [ 6 , 100 , 101 , 104 ]. The effects of appraiser experience on the results of AMSTAR-2 and ROBIS require further evaluation [ 101 , 105 ]. Updates to both tools should address items shown to be prone to individual appraisers’ subjective biases and opinions [ 11 , 100 ]; this may involve modifications of the current domains and signaling questions as well as incorporation of methods to make an appraiser’s judgments more explicit. Future revisions of these tools may also consider the addition of standards for aspects of systematic review development currently lacking (eg, rating overall certainty of evidence, [ 99 ] methods for synthesis without meta-analysis [ 105 ]) and removal of items that assess aspects of reporting that are thoroughly evaluated by PRISMA 2020.
A good understanding of what is required to satisfy the standards of AMSTAR-2 and ROBIS involves study of the accompanying guidance documents written by the tools’ developers; these contain detailed descriptions of each item’s standards. In addition, accurate appraisal of a systematic review with either tool requires training. Most experts recommend independent assessment by at least two appraisers with a process for resolving discrepancies as well as procedures to establish interrater reliability, such as pilot testing, a calibration phase or exercise, and development of predefined decision rules [ 35 , 99 – 101 , 103 , 104 , 106 ]. These methods may, to some extent, address the challenges associated with the diversity in methodological training, subject matter expertise, and experience using the tools that are likely to exist among appraisers.
The standards of AMSTAR, AMSTAR-2, and ROBIS have been used in many methodological studies and epidemiological investigations. However, the increased publication of overviews or umbrella reviews and CPGs has likely been a greater influence on the widening acceptance of these tools. Critical appraisal of the secondary studies considered evidence is essential to the trustworthiness of both the recommendations of CPGs and the conclusions of overviews. Currently both Cochrane [ 55 ] and JBI [ 107 ] recommend AMSTAR-2 and ROBIS in their guidance for authors of overviews or umbrella reviews. However, ROBIS and AMSTAR-2 were released in 2016 and 2017, respectively; thus, to date, limited data have been reported about the uptake of these tools or which of the two may be preferred [ 21 , 106 ]. Currently, in relation to CPGs, AMSTAR-2 appears to be overwhelmingly popular compared to ROBIS. A Google Scholar search of this topic (search terms “AMSTAR 2 AND clinical practice guidelines,” “ROBIS AND clinical practice guidelines” 13 May 2022) found 12,700 hits for AMSTAR-2 and 1,280 for ROBIS. The apparent greater appeal of AMSTAR-2 may relate to its longer track record given the original version of the tool was in use for 10 years prior to its update in 2017.
Barriers to the uptake of AMSTAR-2 and ROBIS include the real or perceived time and resources necessary to complete the items they include and appraisers’ confidence in their own ratings [ 104 ]. Reports from comparative studies available to date indicate that appraisers find AMSTAR-2 questions, responses, and guidance to be clearer and simpler compared with ROBIS [ 11 , 101 , 104 , 105 ]. This suggests that for appraisal of intervention systematic reviews, AMSTAR-2 may be a more practical tool than ROBIS, especially for novice appraisers [ 101 , 103 – 105 ]. The unique characteristics of each tool, as well as their potential advantages and disadvantages, should be taken into consideration when deciding which tool should be used for an appraisal of a systematic review. In addition, the choice of one or the other may depend on how the results of an appraisal will be used; for example, a peer reviewer’s appraisal of a single manuscript versus an appraisal of multiple systematic reviews in an overview or umbrella review, CPG, or systematic methodological study.
Authors of overviews and CPGs report results of AMSTAR-2 and ROBIS appraisals for each of the systematic reviews they include as evidence. Ideally, an independent judgment of their appraisals can be made by the end users of overviews and CPGs; however, most stakeholders, including clinicians, are unlikely to have a sophisticated understanding of these tools. Nevertheless, they should at least be aware that AMSTAR-2 and ROBIS ratings reported in overviews and CPGs may be inaccurate because the tools are not applied as intended by their developers. This can result from inadequate training of the overview or CPG authors who perform the appraisals, or to modifications of the appraisal tools imposed by them. The potential variability in overall confidence and RoB ratings highlights why appraisers applying these tools need to support their judgments with explicit documentation; this allows readers to judge for themselves whether they agree with the criteria used by appraisers [ 4 , 108 ]. When these judgments are explicit, the underlying rationale used when applying these tools can be assessed [ 109 ].
Theoretically, we would expect an association of AMSTAR-2 with improved methodological rigor and an association of ROBIS with lower RoB in recent systematic reviews compared to those published before 2017. To our knowledge, this has not yet been demonstrated; however, like reports about the actual uptake of these tools, time will tell. Additional data on user experience is also needed to further elucidate the practical challenges and methodological nuances encountered with the application of these tools. This information could potentially inform the creation of unifying criteria to guide and standardize the appraisal of evidence syntheses [ 109 ].
Complete reporting is essential for users to establish the trustworthiness and applicability of a systematic review’s findings. Efforts to standardize and improve the reporting of systematic reviews resulted in the 2009 publication of the PRISMA statement [ 92 ] with its accompanying explanation and elaboration document [ 110 ]. This guideline was designed to help authors prepare a complete and transparent report of their systematic review. In addition, adherence to PRISMA is often used to evaluate the thoroughness of reporting of published systematic reviews [ 111 ]. The updated version, PRISMA 2020 [ 93 ], and its guidance document [ 112 ] were published in 2021. Items on the original and updated versions of PRISMA are organized by the six basic review components they address (title, abstract, introduction, methods, results, discussion). The PRISMA 2020 update is a considerably expanded version of the original; it includes standards and examples for the 27 original and 13 additional reporting items that capture methodological advances and may enhance the replicability of reviews [ 113 ].
The original PRISMA statement fostered the development of various PRISMA extensions (Table 3.3 ). These include reporting guidance for scoping reviews and reviews of diagnostic test accuracy and for intervention reviews that report on the following: harms outcomes, equity issues, the effects of acupuncture, the results of network meta-analyses and analyses of individual participant data. Detailed reporting guidance for specific systematic review components (abstracts, protocols, literature searches) is also available.
PRISMA extensions
PRISMA for systematic reviews with a focus on health equity [ ] | PRISMA-E | 2012 | |
Reporting systematic reviews in journal and conference abstracts [ ] | PRISMA for Abstracts | 2015; 2020 | |
PRISMA for systematic review protocols [ ] | PRISMA-P | 2015 | |
PRISMA for Network Meta-Analyses [ ] | PRISMA-NMA | 2015 | |
PRISMA for Individual Participant Data [ ] | PRISMA-IPD | 2015 | |
PRISMA for reviews including harms outcomes [ ] | PRISMA-Harms | 2016 | |
PRISMA for diagnostic test accuracy [ ] | PRISMA-DTA | 2018 | |
PRISMA for scoping reviews [ ] | PRISMA-ScR | 2018 | |
PRISMA for acupuncture [ ] | PRISMA-A | 2019 | |
PRISMA for reporting literature searches [ ] | PRISMA-S | 2021 |
PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses
a Note the abstract reporting checklist is now incorporated into PRISMA 2020 [ 93 ]
The 2009 PRISMA standards [ 92 ] for reporting have been widely endorsed by authors, journals, and EBM-related organizations. We anticipate the same for PRISMA 2020 [ 93 ] given its co-publication in multiple high-impact journals. However, to date, there is a lack of strong evidence for an association between improved systematic review reporting and endorsement of PRISMA 2009 standards [ 43 , 111 ]. Most journals require a PRISMA checklist accompany submissions of systematic review manuscripts. However, the accuracy of information presented on these self-reported checklists is not necessarily verified. It remains unclear which strategies (eg, authors’ self-report of checklists, peer reviewer checks) might improve adherence to the PRISMA reporting standards; in addition, the feasibility of any potentially effective strategies must be taken into consideration given the structure and limitations of current research and publication practices [ 124 ].
Misunderstanding of the roles of these tools and their misapplication may be widespread problems. PRISMA 2020 is a reporting guideline that is most beneficial if consulted when developing a review as opposed to merely completing a checklist when submitting to a journal; at that point, the review is finished, with good or bad methodological choices. However, PRISMA checklists evaluate how completely an element of review conduct was reported, but do not evaluate the caliber of conduct or performance of a review. Thus, review authors and readers should not think that a rigorous systematic review can be produced by simply following the PRISMA 2020 guidelines. Similarly, it is important to recognize that AMSTAR-2 and ROBIS are tools to evaluate the conduct of a review but do not substitute for conceptual methodological guidance. In addition, they are not intended to be simple checklists. In fact, they have the potential for misuse or abuse if applied as such; for example, by calculating a total score to make a judgment about a review’s overall confidence or RoB. Proper selection of a response for the individual items on AMSTAR-2 and ROBIS requires training or at least reference to their accompanying guidance documents.
Not surprisingly, it has been shown that compliance with the PRISMA checklist is not necessarily associated with satisfying the standards of ROBIS [ 125 ]. AMSTAR-2 and ROBIS were not available when PRISMA 2009 was developed; however, they were considered in the development of PRISMA 2020 [ 113 ]. Therefore, future studies may show a positive relationship between fulfillment of PRISMA 2020 standards for reporting and meeting the standards of tools evaluating methodological quality and RoB.
Choice of an appropriate tool for the evaluation of a systematic review first involves identification of the underlying construct to be assessed. For systematic reviews of interventions, recommended tools include AMSTAR-2 and ROBIS for appraisal of conduct and PRISMA 2020 for completeness of reporting. All three tools were developed rigorously and provide easily accessible and detailed user guidance, which is necessary for their proper application and interpretation. When considering a manuscript for publication, training in these tools can sensitize peer reviewers and editors to major issues that may affect the review’s trustworthiness and completeness of reporting. Judgment of the overall certainty of a body of evidence and formulation of recommendations rely, in part, on AMSTAR-2 or ROBIS appraisals of systematic reviews. Therefore, training on the application of these tools is essential for authors of overviews and developers of CPGs. Peer reviewers and editors considering an overview or CPG for publication must hold their authors to a high standard of transparency regarding both the conduct and reporting of these appraisals.
Many authors, peer reviewers, and editors erroneously equate fulfillment of the items on the PRISMA checklist with superior methodological rigor. For direction on methodology, we refer them to available resources that provide comprehensive conceptual guidance [ 59 , 60 ] as well as primers with basic step-by-step instructions [ 1 , 126 , 127 ]. This section is intended to complement study of such resources by facilitating use of AMSTAR-2 and ROBIS, tools specifically developed to evaluate methodological rigor of systematic reviews. These tools are widely accepted by methodologists; however, in the general medical literature, they are not uniformly selected for the critical appraisal of systematic reviews [ 88 , 96 ].
To enable their uptake, Table 4.1 links review components to the corresponding appraisal tool items. Expectations of AMSTAR-2 and ROBIS are concisely stated, and reasoning provided.
Systematic review components linked to appraisal with AMSTAR-2 and ROBIS a
Table | Table | |||
Methods for study selection | #5 | #2.5 | All three components must be done in duplicate, and methods fully described. | Helps to mitigate CoI and bias; also may improve accuracy. |
Methods for data extraction | #6 | #3.1 | ||
Methods for RoB assessment | NA | #3.5 | ||
Study description | #8 | #3.2 | Research design features, components of research question (eg, PICO), setting, funding sources. | Allows readers to understand the individual studies in detail. |
Sources of funding | #10 | NA | Identified for all included studies. | Can reveal CoI or bias. |
Publication bias | #15* | #4.5 | Explored, diagrammed, and discussed. | Publication and other selective reporting biases are major threats to the validity of systematic reviews. |
Author CoI | #16 | NA | Disclosed, with management strategies described. | If CoI is identified, management strategies must be described to ensure confidence in the review. |
CoI conflict of interest, MA meta-analysis, NA not addressed, PICO participant, intervention, comparison, outcome, PRISMA-P Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols, RoB risk of bias
a Components shown in bold are chosen for elaboration in Part 4 for one (or both) of two reasons: 1) the component has been identified as potentially problematic for systematic review authors; and/or 2) the component is evaluated by standards of an AMSTAR-2 “critical” domain
b Critical domains of AMSTAR-2 are indicated by *
Issues involved in meeting the standards for seven review components (identified in bold in Table 4.1 ) are addressed in detail. These were chosen for elaboration for one (or both) of two reasons: 1) the component has been identified as potentially problematic for systematic review authors based on consistent reports of their frequent AMSTAR-2 or ROBIS deficiencies [ 9 , 11 , 15 , 88 , 128 , 129 ]; and/or 2) the review component is judged by standards of an AMSTAR-2 “critical” domain. These have the greatest implications for how a systematic review will be appraised: if standards for any one of these critical domains are not met, the review is rated as having “critically low confidence.”
Specific and unambiguous research questions may have more value for reviews that deal with hypothesis testing. Mnemonics for the various elements of research questions are suggested by JBI and Cochrane (Table 2.1 ). These prompt authors to consider the specialized methods involved for developing different types of systematic reviews; however, while inclusion of the suggested elements makes a review compliant with a particular review’s methods, it does not necessarily make a research question appropriate. Table 4.2 lists acronyms that may aid in developing the research question. They include overlapping concepts of importance in this time of proliferating reviews of uncertain value [ 130 ]. If these issues are not prospectively contemplated, systematic review authors may establish an overly broad scope, or develop runaway scope allowing them to stray from predefined choices relating to key comparisons and outcomes.
Research question development
Acronym | Meaning |
---|---|
feasible, interesting, novel, ethical, and relevant | |
specific, measurable, attainable, relevant, timely | |
time, outcomes, population, intervention, context, study design, plus (effect) moderators |
a Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Hulley SB, Cummings SR, Browner WS, editors. Designing clinical research: an epidemiological approach; 4th edn. Lippincott Williams & Wilkins; 2007. p. 14–22
b Doran, GT. There’s a S.M.A.R.T. way to write management’s goals and objectives. Manage Rev. 1981;70:35-6.
c Johnson BT, Hennessy EA. Systematic reviews and meta-analyses in the health sciences: best practice methods for research syntheses. Soc Sci Med. 2019;233:237–51
Once a research question is established, searching on registry sites and databases for existing systematic reviews addressing the same or a similar topic is necessary in order to avoid contributing to research waste [ 131 ]. Repeating an existing systematic review must be justified, for example, if previous reviews are out of date or methodologically flawed. A full discussion on replication of intervention systematic reviews, including a consensus checklist, can be found in the work of Tugwell and colleagues [ 84 ].
Protocol development is considered a core component of systematic reviews [ 125 , 126 , 132 ]. Review protocols may allow researchers to plan and anticipate potential issues, assess validity of methods, prevent arbitrary decision-making, and minimize bias that can be introduced by the conduct of the review. Registration of a protocol that allows public access promotes transparency of the systematic review’s methods and processes and reduces the potential for duplication [ 132 ]. Thinking early and carefully about all the steps of a systematic review is pragmatic and logical and may mitigate the influence of the authors’ prior knowledge of the evidence [ 133 ]. In addition, the protocol stage is when the scope of the review can be carefully considered by authors, reviewers, and editors; this may help to avoid production of overly ambitious reviews that include excessive numbers of comparisons and outcomes or are undisciplined in their study selection.
An association with attainment of AMSTAR standards in systematic reviews with published prospective protocols has been reported [ 134 ]. However, completeness of reporting does not seem to be different in reviews with a protocol compared to those without one [ 135 ]. PRISMA-P [ 116 ] and its accompanying elaboration and explanation document [ 136 ] can be used to guide and assess the reporting of protocols. A final version of the review should fully describe any protocol deviations. Peer reviewers may compare the submitted manuscript with any available pre-registered protocol; this is required if AMSTAR-2 or ROBIS are used for critical appraisal.
There are multiple options for the recording of protocols (Table 4.3 ). Some journals will peer review and publish protocols. In addition, many online sites offer date-stamped and publicly accessible protocol registration. Some of these are exclusively for protocols of evidence syntheses; others are less restrictive and offer researchers the capacity for data storage, sharing, and other workflow features. These sites document protocol details to varying extents and have different requirements [ 137 ]. The most popular site for systematic reviews, the International Prospective Register of Systematic Reviews (PROSPERO), for example, only registers reviews that report on an outcome with direct relevance to human health. The PROSPERO record documents protocols for all types of reviews except literature and scoping reviews. Of note, PROSPERO requires authors register their review protocols prior to any data extraction [ 133 , 138 ]. The electronic records of most of these registry sites allow authors to update their protocols and facilitate transparent tracking of protocol changes, which are not unexpected during the progress of the review [ 139 ].
Options for protocol registration of evidence syntheses
BMJ Open | |
BioMed Central | |
JMIR Research Protocols | |
World Journal of Meta-analysis | |
Cochrane | |
JBI | |
PROSPERO | |
Research Registry- Registry of Systematic Reviews/Meta-Analyses | |
International Platform of Registered Systematic Review and Meta-analysis Protocols (INPLASY) | |
Center for Open Science | |
Protocols.io | |
Figshare | |
Open Science Framework | |
Zenodo |
a Authors are advised to contact their target journal regarding submission of systematic review protocols
b Registration is restricted to approved review projects
c The JBI registry lists review projects currently underway by JBI-affiliated entities. These records include a review’s title, primary author, research question, and PICO elements. JBI recommends that authors register eligible protocols with PROSPERO
d See Pieper and Rombey [ 137 ] for detailed characteristics of these five registries
e See Pieper and Rombey [ 137 ] for other systematic review data repository options
For most systematic reviews, broad inclusion of study designs is recommended [ 126 ]. This may allow comparison of results between contrasting study design types [ 126 ]. Certain study designs may be considered preferable depending on the type of review and nature of the research question. However, prevailing stereotypes about what each study design does best may not be accurate. For example, in systematic reviews of interventions, randomized designs are typically thought to answer highly specific questions while non-randomized designs often are expected to reveal greater information about harms or real-word evidence [ 126 , 140 , 141 ]. This may be a false distinction; randomized trials may be pragmatic [ 142 ], they may offer important (and more unbiased) information on harms [ 143 ], and data from non-randomized trials may not necessarily be more real-world-oriented [ 144 ].
Moreover, there may not be any available evidence reported by RCTs for certain research questions; in some cases, there may not be any RCTs or NRSI. When the available evidence is limited to case reports and case series, it is not possible to test hypotheses nor provide descriptive estimates or associations; however, a systematic review of these studies can still offer important insights [ 81 , 145 ]. When authors anticipate that limited evidence of any kind may be available to inform their research questions, a scoping review can be considered. Alternatively, decisions regarding inclusion of indirect as opposed to direct evidence can be addressed during protocol development [ 146 ]. Including indirect evidence at an early stage of intervention systematic review development allows authors to decide if such studies offer any additional and/or different understanding of treatment effects for their population or comparison of interest. Issues of indirectness of included studies are accounted for later in the process, during determination of the overall certainty of evidence (see Part 5 for details).
Both AMSTAR-2 and ROBIS require systematic and comprehensive searches for evidence. This is essential for any systematic review. Both tools discourage search restrictions based on language and publication source. Given increasing globalism in health care, the practice of including English-only literature should be avoided [ 126 ]. There are many examples in which language bias (different results in studies published in different languages) has been documented [ 147 , 148 ]. This does not mean that all literature, in all languages, is equally trustworthy [ 148 ]; however, the only way to formally probe for the potential of such biases is to consider all languages in the initial search. The gray literature and a search of trials may also reveal important details about topics that would otherwise be missed [ 149 – 151 ]. Again, inclusiveness will allow review authors to investigate whether results differ in gray literature and trials [ 41 , 151 – 153 ].
Authors should make every attempt to complete their review within one year as that is the likely viable life of a search. (1) If that is not possible, the search should be updated close to the time of completion [ 154 ]. Different research topics may warrant less of a delay, for example, in rapidly changing fields (as in the case of the COVID-19 pandemic), even one month may radically change the available evidence.
AMSTAR-2 requires authors to provide references for any studies excluded at the full text phase of study selection along with reasons for exclusion; this allows readers to feel confident that all relevant literature has been considered for inclusion and that exclusions are defensible.
The design of the studies included in a systematic review (eg, RCT, cohort, case series) should not be equated with appraisal of its RoB. To meet AMSTAR-2 and ROBIS standards, systematic review authors must examine RoB issues specific to the design of each primary study they include as evidence. It is unlikely that a single RoB appraisal tool will be suitable for all research designs. In addition to tools for randomized and non-randomized studies, specific tools are available for evaluation of RoB in case reports and case series [ 82 ] and single-case experimental designs [ 155 , 156 ]. Note the RoB tools selected must meet the standards of the appraisal tool used to judge the conduct of the review. For example, AMSTAR-2 identifies four sources of bias specific to RCTs and NRSI that must be addressed by the RoB tool(s) chosen by the review authors. The Cochrane RoB-2 [ 157 ] tool for RCTs and ROBINS-I [ 158 ] for NRSI for RoB assessment meet the AMSTAR-2 standards. Appraisers on the review team should not modify any RoB tool without complete transparency and acknowledgment that they have invalidated the interpretation of the tool as intended by its developers [ 159 ]. Conduct of RoB assessments is not addressed AMSTAR-2; to meet ROBIS standards, two independent reviewers should complete RoB assessments of included primary studies.
Implications of the RoB assessments must be explicitly discussed and considered in the conclusions of the review. Discussion of the overall RoB of included studies may consider the weight of the studies at high RoB, the importance of the sources of bias in the studies being summarized, and if their importance differs in relationship to the outcomes reported. If a meta-analysis is performed, serious concerns for RoB of individual studies should be accounted for in these results as well. If the results of the meta-analysis for a specific outcome change when studies at high RoB are excluded, readers will have a more accurate understanding of this body of evidence. However, while investigating the potential impact of specific biases is a useful exercise, it is important to avoid over-interpretation, especially when there are sparse data.
Syntheses of quantitative data reported by primary studies are broadly categorized as one of two types: meta-analysis, and synthesis without meta-analysis (Table 4.4 ). Before deciding on one of these methods, authors should seek methodological advice about whether reported data can be transformed or used in other ways to provide a consistent effect measure across studies [ 160 , 161 ].
Common methods for quantitative synthesis
Aggregate data Individual participant data | Weighted average of effect estimates | Pairwise comparisons of effect estimates, CI Overall effect estimate, CI, value Evaluation of heterogeneity | Forest plot with summary statistic for average effect estimate |
Network | Variable | The interventions, which are compared directly indirectly | Network diagram or graph, tabular presentations |
Comparisons of relative effects between any pair of interventions | Effect estimates for intervention pairings | ||
Summary relative effects for pair-wise comparisons with evaluations of inconsistency and heterogeneity | Forest plot, other methods | ||
Treatment rankings (ie, probability that an intervention is among the best options) | Rankogram plot | ||
Summarizing effect estimates from separate studies (without combination that would provide an average effect estimate) | Range and distribution of observed effects such as median, interquartile range, range | Box-and-whisker plot, bubble plot Forest plot (without summary effect estimate) | |
Combining values | Combined value, number of studies | Albatross plot (study sample size against values per outcome) | |
Vote counting by direction of effect (eg, favors intervention over the comparator) | Proportion of studies with an effect in the direction of interest, CI, value | Harvest plot, effect direction plot |
CI confidence interval (or credible interval, if analysis is done in Bayesian framework)
a See text for descriptions of the types of data combined in each of these approaches
b See Additional File 4 for guidance on the structure and presentation of forest plots
c General approach is similar to aggregate data meta-analysis but there are substantial differences relating to data collection and checking and analysis [ 162 ]. This approach to syntheses is applicable to intervention, diagnostic, and prognostic systematic reviews [ 163 ]
d Examples include meta-regression, hierarchical and multivariate approaches [ 164 ]
e In-depth guidance and illustrations of these methods are provided in Chapter 12 of the Cochrane Handbook [ 160 ]
Systematic reviews that employ meta-analysis should not be referred to simply as “meta-analyses.” The term meta-analysis strictly refers to a specific statistical technique used when study effect estimates and their variances are available, yielding a quantitative summary of results. In general, methods for meta-analysis involve use of a weighted average of effect estimates from two or more studies. If considered carefully, meta-analysis increases the precision of the estimated magnitude of effect and can offer useful insights about heterogeneity and estimates of effects. We refer to standard references for a thorough introduction and formal training [ 165 – 167 ].
There are three common approaches to meta-analysis in current health care–related systematic reviews (Table 4.4 ). Aggregate meta-analyses is the most familiar to authors of evidence syntheses and their end users. This standard meta-analysis combines data on effect estimates reported by studies that investigate similar research questions involving direct comparisons of an intervention and comparator. Results of these analyses provide a single summary intervention effect estimate. If the included studies in a systematic review measure an outcome differently, their reported results may be transformed to make them comparable [ 161 ]. Forest plots visually present essential information about the individual studies and the overall pooled analysis (see Additional File 4 for details).
Less familiar and more challenging meta-analytical approaches used in secondary research include individual participant data (IPD) and network meta-analyses (NMA); PRISMA extensions provide reporting guidelines for both [ 117 , 118 ]. In IPD, the raw data on each participant from each eligible study are re-analyzed as opposed to the study-level data analyzed in aggregate data meta-analyses [ 168 ]. This may offer advantages, including the potential for limiting concerns about bias and allowing more robust analyses [ 163 ]. As suggested by the description in Table 4.4 , NMA is a complex statistical approach. It combines aggregate data [ 169 ] or IPD [ 170 ] for effect estimates from direct and indirect comparisons reported in two or more studies of three or more interventions. This makes it a potentially powerful statistical tool; while multiple interventions are typically available to treat a condition, few have been evaluated in head-to-head trials [ 171 ]. Both IPD and NMA facilitate a broader scope, and potentially provide more reliable and/or detailed results; however, compared with standard aggregate data meta-analyses, their methods are more complicated, time-consuming, and resource-intensive, and they have their own biases, so one needs sufficient funding, technical expertise, and preparation to employ them successfully [ 41 , 172 , 173 ].
Several items in AMSTAR-2 and ROBIS address meta-analysis; thus, understanding the strengths, weaknesses, assumptions, and limitations of methods for meta-analyses is important. According to the standards of both tools, plans for a meta-analysis must be addressed in the review protocol, including reasoning, description of the type of quantitative data to be synthesized, and the methods planned for combining the data. This should not consist of stock statements describing conventional meta-analysis techniques; rather, authors are expected to anticipate issues specific to their research questions. Concern for the lack of training in meta-analysis methods among systematic review authors cannot be overstated. For those with training, the use of popular software (eg, RevMan [ 174 ], MetaXL [ 175 ], JBI SUMARI [ 176 ]) may facilitate exploration of these methods; however, such programs cannot substitute for the accurate interpretation of the results of meta-analyses, especially for more complex meta-analytical approaches.
There are varied reasons a meta-analysis may not be appropriate or desirable [ 160 , 161 ]. Syntheses that informally use statistical methods other than meta-analysis are variably referred to as descriptive, narrative, or qualitative syntheses or summaries; these terms are also applied to syntheses that make no attempt to statistically combine data from individual studies. However, use of such imprecise terminology is discouraged; in order to fully explore the results of any type of synthesis, some narration or description is needed to supplement the data visually presented in tabular or graphic forms [ 63 , 177 ]. In addition, the term “qualitative synthesis” is easily confused with a synthesis of qualitative data in a qualitative or mixed methods review. “Synthesis without meta-analysis” is currently the preferred description of other ways to combine quantitative data from two or more studies. Use of this specific terminology when referring to these types of syntheses also implies the application of formal methods (Table 4.4 ).
Methods for syntheses without meta-analysis involve structured presentations of the data in any tables and plots. In comparison to narrative descriptions of each study, these are designed to more effectively and transparently show patterns and convey detailed information about the data; they also allow informal exploration of heterogeneity [ 178 ]. In addition, acceptable quantitative statistical methods (Table 4.4 ) are formally applied; however, it is important to recognize these methods have significant limitations for the interpretation of the effectiveness of an intervention [ 160 ]. Nevertheless, when meta-analysis is not possible, the application of these methods is less prone to bias compared with an unstructured narrative description of included studies [ 178 , 179 ].
Vote counting is commonly used in systematic reviews and involves a tally of studies reporting results that meet some threshold of importance applied by review authors. Until recently, it has not typically been identified as a method for synthesis without meta-analysis. Guidance on an acceptable vote counting method based on direction of effect is currently available [ 160 ] and should be used instead of narrative descriptions of such results (eg, “more than half the studies showed improvement”; “only a few studies reported adverse effects”; “7 out of 10 studies favored the intervention”). Unacceptable methods include vote counting by statistical significance or magnitude of effect or some subjective rule applied by the authors.
AMSTAR-2 and ROBIS standards do not explicitly address conduct of syntheses without meta-analysis, although AMSTAR-2 items 13 and 14 might be considered relevant. Guidance for the complete reporting of syntheses without meta-analysis for systematic reviews of interventions is available in the Synthesis without Meta-analysis (SWiM) guideline [ 180 ] and methodological guidance is available in the Cochrane Handbook [ 160 , 181 ].
Familiarity with AMSTAR-2 and ROBIS makes sense for authors of systematic reviews as these appraisal tools will be used to judge their work; however, training is necessary for authors to truly appreciate and apply methodological rigor. Moreover, judgment of the potential contribution of a systematic review to the current knowledge base goes beyond meeting the standards of AMSTAR-2 and ROBIS. These tools do not explicitly address some crucial concepts involved in the development of a systematic review; this further emphasizes the need for author training.
We recommend that systematic review authors incorporate specific practices or exercises when formulating a research question at the protocol stage, These should be designed to raise the review team’s awareness of how to prevent research and resource waste [ 84 , 130 ] and to stimulate careful contemplation of the scope of the review [ 30 ]. Authors’ training should also focus on justifiably choosing a formal method for the synthesis of quantitative and/or qualitative data from primary research; both types of data require specific expertise. For typical reviews that involve syntheses of quantitative data, statistical expertise is necessary, initially for decisions about appropriate methods, [ 160 , 161 ] and then to inform any meta-analyses [ 167 ] or other statistical methods applied [ 160 ].
Report of an overall certainty of evidence assessment in a systematic review is an important new reporting standard of the updated PRISMA 2020 guidelines [ 93 ]. Systematic review authors are well acquainted with assessing RoB in individual primary studies, but much less familiar with assessment of overall certainty across an entire body of evidence. Yet a reliable way to evaluate this broader concept is now recognized as a vital part of interpreting the evidence.
Historical systems for rating evidence are based on study design and usually involve hierarchical levels or classes of evidence that use numbers and/or letters to designate the level/class. These systems were endorsed by various EBM-related organizations. Professional societies and regulatory groups then widely adopted them, often with modifications for application to the available primary research base in specific clinical areas. In 2002, a report issued by the AHRQ identified 40 systems to rate quality of a body of evidence [ 182 ]. A critical appraisal of systems used by prominent health care organizations published in 2004 revealed limitations in sensibility, reproducibility, applicability to different questions, and usability to different end users [ 183 ]. Persistent use of hierarchical rating schemes to describe overall quality continues to complicate the interpretation of evidence. This is indicated by recent reports of poor interpretability of systematic review results by readers [ 184 – 186 ] and misleading interpretations of the evidence related to the “spin” systematic review authors may put on their conclusions [ 50 , 187 ].
Recognition of the shortcomings of hierarchical rating systems raised concerns that misleading clinical recommendations could result even if based on a rigorous systematic review. In addition, the number and variability of these systems were considered obstacles to quick and accurate interpretations of the evidence by clinicians, patients, and policymakers [ 183 ]. These issues contributed to the development of the GRADE approach. An international working group, that continues to actively evaluate and refine it, first introduced GRADE in 2004 [ 188 ]. Currently more than 110 organizations from 19 countries around the world have endorsed or are using GRADE [ 189 ].
GRADE offers a consistent and sensible approach for two separate processes: rating the overall certainty of a body of evidence and the strength of recommendations. The former is the expected conclusion of a systematic review, while the latter is pertinent to the development of CPGs. As such, GRADE provides a mechanism to bridge the gap from evidence synthesis to application of the evidence for informed clinical decision-making [ 27 , 190 ]. We briefly examine the GRADE approach but only as it applies to rating overall certainty of evidence in systematic reviews.
In GRADE, use of “certainty” of a body of evidence is preferred over the term “quality.” [ 191 ] Certainty refers to the level of confidence systematic review authors have that, for each outcome, an effect estimate represents the true effect. The GRADE approach to rating confidence in estimates begins with identifying the study type (RCT or NRSI) and then systematically considers criteria to rate the certainty of evidence up or down (Table 5.1 ).
GRADE criteria for rating certainty of evidence
[ ] | |
---|---|
Risk of bias [ ] | Large magnitude of effect |
Imprecision [ ] | Dose–response gradient |
Inconsistency [ ] | All residual confounding would decrease magnitude of effect (in situations with an effect) |
Indirectness [ ] | |
Publication bias [ ] |
a Applies to randomized studies
b Applies to non-randomized studies
This process results in assignment of one of the four GRADE certainty ratings to each outcome; these are clearly conveyed with the use of basic interpretation symbols (Table 5.2 ) [ 192 ]. Notably, when multiple outcomes are reported in a systematic review, each outcome is assigned a unique certainty rating; thus different levels of certainty may exist in the body of evidence being examined.
GRADE certainty ratings and their interpretation symbols a
⊕ ⊕ ⊕ ⊕ High: We are very confident that the true effect lies close to that of the estimate of the effect |
⊕ ⊕ ⊕ Moderate: We are moderately confident in the effect estimate: the true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different |
⊕ ⊕ Low: Our confidence in the effect estimate is limited: the true effect may be substantially different from the estimate of the effect |
⊕ Very low: We have very little confidence in the effect estimate: the true effect is likely to be substantially different from the estimate of effect |
a From the GRADE Handbook [ 192 ]
GRADE’s developers acknowledge some subjectivity is involved in this process [ 193 ]. In addition, they emphasize that both the criteria for rating evidence up and down (Table 5.1 ) as well as the four overall certainty ratings (Table 5.2 ) reflect a continuum as opposed to discrete categories [ 194 ]. Consequently, deciding whether a study falls above or below the threshold for rating up or down may not be straightforward, and preliminary overall certainty ratings may be intermediate (eg, between low and moderate). Thus, the proper application of GRADE requires systematic review authors to take an overall view of the body of evidence and explicitly describe the rationale for their final ratings.
Outcomes important to the individuals who experience the problem of interest maintain a prominent role throughout the GRADE process [ 191 ]. These outcomes must inform the research questions (eg, PICO [population, intervention, comparator, outcome]) that are specified a priori in a systematic review protocol. Evidence for these outcomes is then investigated and each critical or important outcome is ultimately assigned a certainty of evidence as the end point of the review. Notably, limitations of the included studies have an impact at the outcome level. Ultimately, the certainty ratings for each outcome reported in a systematic review are considered by guideline panels. They use a different process to formulate recommendations that involves assessment of the evidence across outcomes [ 201 ]. It is beyond our scope to describe the GRADE process for formulating recommendations; however, it is critical to understand how these two outcome-centric concepts of certainty of evidence in the GRADE framework are related and distinguished. An in-depth illustration using examples from recently published evidence syntheses and CPGs is provided in Additional File 5 A (Table AF5A-1).
The GRADE approach is applicable irrespective of whether the certainty of the primary research evidence is high or very low; in some circumstances, indirect evidence of higher certainty may be considered if direct evidence is unavailable or of low certainty [ 27 ]. In fact, most interventions and outcomes in medicine have low or very low certainty of evidence based on GRADE and there seems to be no major improvement over time [ 202 , 203 ]. This is still a very important (even if sobering) realization for calibrating our understanding of medical evidence. A major appeal of the GRADE approach is that it offers a common framework that enables authors of evidence syntheses to make complex judgments about evidence certainty and to convey these with unambiguous terminology. This prevents some common mistakes made by review authors, including overstating results (or under-reporting harms) [ 187 ] and making recommendations for treatment. This is illustrated in Table AF5A-2 (Additional File 5 A), which compares the concluding statements made about overall certainty in a systematic review with and without application of the GRADE approach.
Theoretically, application of GRADE should improve consistency of judgments about certainty of evidence, both between authors and across systematic reviews. In one empirical evaluation conducted by the GRADE Working Group, interrater reliability of two individual raters assessing certainty of the evidence for a specific outcome increased from ~ 0.3 without using GRADE to ~ 0.7 by using GRADE [ 204 ]. However, others report variable agreement among those experienced in GRADE assessments of evidence certainty [ 190 ]. Like any other tool, GRADE requires training in order to be properly applied. The intricacies of the GRADE approach and the necessary subjectivity involved suggest that improving agreement may require strict rules for its application; alternatively, use of general guidance and consensus among review authors may result in less consistency but provide important information for the end user [ 190 ].
Simply invoking “the GRADE approach” does not automatically ensure GRADE methods were employed by authors of a systematic review (or developers of a CPG). Table 5.3 lists the criteria the GRADE working group has established for this purpose. These criteria highlight the specific terminology and methods that apply to rating the certainty of evidence for outcomes reported in a systematic review [ 191 ], which is different from rating overall certainty across outcomes considered in the formulation of recommendations [ 205 ]. Modifications of standard GRADE methods and terminology are discouraged as these may detract from GRADE’s objectives to minimize conceptual confusion and maximize clear communication [ 206 ].
Criteria for using GRADE in a systematic review a
1. The certainty in the evidence (also known as quality of evidence or confidence in the estimates) should be defined consistently with the definitions used by the GRADE Working Group. |
2. Explicit consideration should be given to each of the GRADE domains for assessing the certainty in the evidence (although different terminology may be used). |
3. The overall certainty in the evidence should be assessed for each important outcome using four or three categories (such as high, moderate, low and/or very low) and definitions for each category that are consistent with the definitions used by the GRADE Working Group. |
4. Evidence summaries … should be used as the basis for judgments about the certainty in the evidence. |
a Adapted from the GRADE working group [ 206 ]; this list does not contain the additional criteria that apply to the development of a clinical practice guideline
Nevertheless, GRADE is prone to misapplications [ 207 , 208 ], which can distort a systematic review’s conclusions about the certainty of evidence. Systematic review authors without proper GRADE training are likely to misinterpret the terms “quality” and “grade” and to misunderstand the constructs assessed by GRADE versus other appraisal tools. For example, review authors may reference the standard GRADE certainty ratings (Table 5.2 ) to describe evidence for their outcome(s) of interest. However, these ratings are invalidated if authors omit or inadequately perform RoB evaluations of each included primary study. Such deficiencies in RoB assessments are unacceptable but not uncommon, as reported in methodological studies of systematic reviews and overviews [ 104 , 186 , 209 , 210 ]. GRADE ratings are also invalidated if review authors do not formally address and report on the other criteria (Table 5.1 ) necessary for a GRADE certainty rating.
Other caveats pertain to application of a GRADE certainty of evidence rating in various types of evidence syntheses. Current adaptations of GRADE are described in Additional File 5 B and included on Table 6.3 , which is introduced in the next section.
Concise Guide to best practices for evidence syntheses, version 1.0 a
Cochrane , JBI | Cochrane, JBI | Cochrane | Cochrane, JBI | JBI | JBI | JBI | Cochrane, JBI | JBI | |
Protocol | PRISMA-P [ ] | PRISMA-P | PRISMA-P | PRISMA-P | PRISMA-P | PRISMA-P | PRISMA-P | PRISMA-P | PRISMA-P |
Systematic review | PRISMA 2020 [ ] | PRISMA-DTA [ ] | PRISMA 2020 | eMERGe [ ] ENTREQ [ ] | PRISMA 2020 | PRISMA 2020 | PRISMA 2020 | PRIOR [ ] | PRISMA-ScR [ ] |
Synthesis without MA | SWiM [ ] | PRISMA-DTA [ ] | SWiM | eMERGe [ ] ENTREQ [ ] | SWiM | SWiM | SWiM | PRIOR [ ] | |
For RCTs: Cochrane RoB2 [ ] For NRSI: ROBINS-I [ ] Other primary research | QUADAS-2[ ] | Factor review QUIPS [ ] Model review PROBAST [ ] | CASP qualitative checklist [ ] JBI Critical Appraisal Checklist [ ] | JBI checklist for studies reporting prevalence data [ ] | For NRSI: ROBINS-I [ ] Other primary research | COSMIN RoB Checklist [ ] | AMSTAR-2 [ ] or ROBIS [ ] | Not required | |
GRADE [ ] | GRADE adaptation | GRADE adaptation | CERQual [ ] ConQual [ ] | GRADE adaptation | Risk factors | GRADE adaptation | GRADE (for intervention reviews) Risk factors | Not applicable |
AMSTAR A MeaSurement Tool to Assess Systematic Reviews, CASP Critical Appraisal Skills Programme, CERQual Confidence in the Evidence from Reviews of Qualitative research, ConQual Establishing Confidence in the output of Qualitative research synthesis, COSMIN COnsensus-based Standards for the selection of health Measurement Instruments, DTA diagnostic test accuracy, eMERGe meta-ethnography reporting guidance, ENTREQ enhancing transparency in reporting the synthesis of qualitative research, GRADE Grading of Recommendations Assessment, Development and Evaluation, MA meta-analysis, NRSI non-randomized studies of interventions, P protocol, PRIOR Preferred Reporting Items for Overviews of Reviews, PRISMA Preferred Reporting Items for Systematic Reviews and Meta-Analyses, PROBAST Prediction model Risk Of Bias ASsessment Tool, QUADAS quality assessment of studies of diagnostic accuracy included in systematic reviews, QUIPS Quality In Prognosis Studies, RCT randomized controlled trial, RoB risk of bias, ROBINS-I Risk Of Bias In Non-randomised Studies of Interventions, ROBIS Risk of Bias in Systematic Reviews, ScR scoping review, SWiM systematic review without meta-analysis
a Superscript numbers represent citations provided in the main reference list. Additional File 6 lists links to available online resources for the methods and tools included in the Concise Guide
b The MECIR manual [ 30 ] provides Cochrane’s specific standards for both reporting and conduct of intervention systematic reviews and protocols
c Editorial and peer reviewers can evaluate completeness of reporting in submitted manuscripts using these tools. Authors may be required to submit a self-reported checklist for the applicable tools
d The decision flowchart described by Flemming and colleagues [ 223 ] is recommended for guidance on how to choose the best approach to reporting for qualitative reviews
e SWiM was developed for intervention studies reporting quantitative data. However, if there is not a more directly relevant reporting guideline, SWiM may prompt reviewers to consider the important details to report. (Personal Communication via email, Mhairi Campbell, 14 Dec 2022)
f JBI recommends their own tools for the critical appraisal of various quantitative primary study designs included in systematic reviews of intervention effectiveness, prevalence and incidence, and etiology and risk as well as for the critical appraisal of systematic reviews included in umbrella reviews. However, except for the JBI Checklists for studies reporting prevalence data and qualitative research, the development, validity, and reliability of these tools are not well documented
g Studies that are not RCTs or NRSI require tools developed specifically to evaluate their design features. Examples include single case experimental design [ 155 , 156 ] and case reports and series [ 82 ]
h The evaluation of methodological quality of studies included in a synthesis of qualitative research is debatable [ 224 ]. Authors may select a tool appropriate for the type of qualitative synthesis methodology employed. The CASP Qualitative Checklist [ 218 ] is an example of a published, commonly used tool that focuses on assessment of the methodological strengths and limitations of qualitative studies. The JBI Critical Appraisal Checklist for Qualitative Research [ 219 ] is recommended for reviews using a meta-aggregative approach
i Consider including risk of bias assessment of included studies if this information is relevant to the research question; however, scoping reviews do not include an assessment of the overall certainty of a body of evidence
j Guidance available from the GRADE working group [ 225 , 226 ]; also recommend consultation with the Cochrane diagnostic methods group
k Guidance available from the GRADE working group [ 227 ]; also recommend consultation with Cochrane prognostic methods group
l Used for syntheses in reviews with a meta-aggregative approach [ 224 ]
m Chapter 5 in the JBI Manual offers guidance on how to adapt GRADE to prevalence and incidence reviews [ 69 ]
n Janiaud and colleagues suggest criteria for evaluating evidence certainty for meta-analyses of non-randomized studies evaluating risk factors [ 228 ]
o The COSMIN user manual provides details on how to apply GRADE in systematic reviews of measurement properties [ 229 ]
The expected culmination of a systematic review should be a rating of overall certainty of a body of evidence for each outcome reported. The GRADE approach is recommended for making these judgments for outcomes reported in systematic reviews of interventions and can be adapted for other types of reviews. This represents the initial step in the process of making recommendations based on evidence syntheses. Peer reviewers should ensure authors meet the minimal criteria for supporting the GRADE approach when reviewing any evidence synthesis that reports certainty ratings derived using GRADE. Authors and peer reviewers of evidence syntheses unfamiliar with GRADE are encouraged to seek formal training and take advantage of the resources available on the GRADE website [ 211 , 212 ].
Accumulating data in recent years suggest that many evidence syntheses (with or without meta-analysis) are not reliable. This relates in part to the fact that their authors, who are often clinicians, can be overwhelmed by the plethora of ways to evaluate evidence. They tend to resort to familiar but often inadequate, inappropriate, or obsolete methods and tools and, as a result, produce unreliable reviews. These manuscripts may not be recognized as such by peer reviewers and journal editors who may disregard current standards. When such a systematic review is published or included in a CPG, clinicians and stakeholders tend to believe that it is trustworthy. A vicious cycle in which inadequate methodology is rewarded and potentially misleading conclusions are accepted is thus supported. There is no quick or easy way to break this cycle; however, increasing awareness of best practices among all these stakeholder groups, who often have minimal (if any) training in methodology, may begin to mitigate it. This is the rationale for inclusion of Parts 2 through 5 in this guidance document. These sections present core concepts and important methodological developments that inform current standards and recommendations. We conclude by taking a direct and practical approach.
Inconsistent and imprecise terminology used in the context of development and evaluation of evidence syntheses is problematic for authors, peer reviewers and editors, and may lead to the application of inappropriate methods and tools. In response, we endorse use of the basic terms (Table 6.1 ) defined in the PRISMA 2020 statement [ 93 ]. In addition, we have identified several problematic expressions and nomenclature. In Table 6.2 , we compile suggestions for preferred terms less likely to be misinterpreted.
Terms relevant to the reporting of health care–related evidence syntheses a
A review that uses explicit, systematic methods to collate and synthesize findings of studies that address a clearly formulated question. |
The combination of quantitative results of two or more studies. This encompasses meta-analysis of effect estimates and other methods, such as combining values, calculating the range and distribution of observed effects, and vote counting based on the direction of effect. |
A statistical technique used to synthesize results when study effect estimates and their variances are available, yielding a quantitative summary of results. |
An event or measurement collected for participants in a study (such as quality of life, mortality). |
The combination of a point estimate (such as a mean difference, risk ratio or proportion) and a measure of its precision (such as a confidence/credible interval) for a particular outcome. |
A document (paper or electronic) supplying information about a particular study. It could be a journal article, preprint, conference abstract, study register entry, clinical study report, dissertation, unpublished manuscript, government report, or any other document providing relevant information. |
The title or abstract (or both) of a report indexed in a database or website (such as a title or abstract for an article indexed in Medline). Records that refer to the same report (such as the same journal article) are “duplicates”; however, records that refer to reports that are merely similar (such as a similar abstract submitted to two different conferences) should be considered unique. |
An investigation, such as a clinical trial, that includes a defined group of participants and one or more interventions and outcomes. A “study” might have multiple reports. For example, reports could include the protocol, statistical analysis plan, baseline characteristics, results for the primary outcome, results for harms, results for secondary outcomes, and results for additional mediator and moderator analyses. |
a Reproduced from Page and colleagues [ 93 ]
Terminology suggestions for health care–related evidence syntheses
Preferred | Potentially problematic |
---|---|
Evidence synthesis with meta-analysis Systematic review with meta-analysis | Meta-analysis |
Overview or umbrella review | Systematic review of systematic reviews Review of reviews Meta-review |
Randomized | Experimental |
Non-randomized | Observational |
Single case experimental design | Single-subject research N-of-1 design |
Case report or case series | Descriptive study |
Methodological quality | Quality |
Certainty of evidence | Quality of evidence Grade of evidence Level of evidence Strength of evidence |
Qualitative systematic review | Qualitative synthesis |
Synthesis of qualitative data | Qualitative synthesis |
Synthesis without meta-analysis | Narrative synthesis , narrative summary Qualitative synthesis Descriptive synthesis, descriptive summary |
a For example, meta-aggregation, meta-ethnography, critical interpretative synthesis, realist synthesis
b This term may best apply to the synthesis in a mixed methods systematic review in which data from different types of evidence (eg, qualitative, quantitative, economic) are summarized [ 64 ]
We also propose a Concise Guide (Table 6.3 ) that summarizes the methods and tools recommended for the development and evaluation of nine types of evidence syntheses. Suggestions for specific tools are based on the rigor of their development as well as the availability of detailed guidance from their developers to ensure their proper application. The formatting of the Concise Guide addresses a well-known source of confusion by clearly distinguishing the underlying methodological constructs that these tools were designed to assess. Important clarifications and explanations follow in the guide’s footnotes; associated websites, if available, are listed in Additional File 6 .
To encourage uptake of best practices, journal editors may consider adopting or adapting the Concise Guide in their instructions to authors and peer reviewers of evidence syntheses. Given the evolving nature of evidence synthesis methodology, the suggested methods and tools are likely to require regular updates. Authors of evidence syntheses should monitor the literature to ensure they are employing current methods and tools. Some types of evidence syntheses (eg, rapid, economic, methodological) are not included in the Concise Guide; for these, authors are advised to obtain recommendations for acceptable methods by consulting with their target journal.
We encourage the appropriate and informed use of the methods and tools discussed throughout this commentary and summarized in the Concise Guide (Table 6.3 ). However, we caution against their application in a perfunctory or superficial fashion. This is a common pitfall among authors of evidence syntheses, especially as the standards of such tools become associated with acceptance of a manuscript by a journal. Consequently, published evidence syntheses may show improved adherence to the requirements of these tools without necessarily making genuine improvements in their performance.
In line with our main objective, the suggested tools in the Concise Guide address the reliability of evidence syntheses; however, we recognize that the utility of systematic reviews is an equally important concern. An unbiased and thoroughly reported evidence synthesis may still not be highly informative if the evidence itself that is summarized is sparse, weak and/or biased [ 24 ]. Many intervention systematic reviews, including those developed by Cochrane [ 203 ] and those applying GRADE [ 202 ], ultimately find no evidence, or find the evidence to be inconclusive (eg, “weak,” “mixed,” or of “low certainty”). This often reflects the primary research base; however, it is important to know what is known (or not known) about a topic when considering an intervention for patients and discussing treatment options with them.
Alternatively, the frequency of “empty” and inconclusive reviews published in the medical literature may relate to limitations of conventional methods that focus on hypothesis testing; these have emphasized the importance of statistical significance in primary research and effect sizes from aggregate meta-analyses [ 183 ]. It is becoming increasingly apparent that this approach may not be appropriate for all topics [ 130 ]. Development of the GRADE approach has facilitated a better understanding of significant factors (beyond effect size) that contribute to the overall certainty of evidence. Other notable responses include the development of integrative synthesis methods for the evaluation of complex interventions [ 230 , 231 ], the incorporation of crowdsourcing and machine learning into systematic review workflows (eg the Cochrane Evidence Pipeline) [ 2 ], the shift in paradigm to living systemic review and NMA platforms [ 232 , 233 ] and the proposal of a new evidence ecosystem that fosters bidirectional collaborations and interactions among a global network of evidence synthesis stakeholders [ 234 ]. These evolutions in data sources and methods may ultimately make evidence syntheses more streamlined, less duplicative, and more importantly, they may be more useful for timely policy and clinical decision-making; however, that will only be the case if they are rigorously reported and conducted.
We look forward to others’ ideas and proposals for the advancement of methods for evidence syntheses. For now, we encourage dissemination and uptake of the currently accepted best tools and practices for their development and evaluation; at the same time, we stress that uptake of appraisal tools, checklists, and software programs cannot substitute for proper education in the methodology of evidence syntheses and meta-analysis. Authors, peer reviewers, and editors must strive to make accurate and reliable contributions to the present evidence knowledge base; online alerts, upcoming technology, and accessible education may make this more feasible than ever before. Our intention is to improve the trustworthiness of evidence syntheses across disciplines, topics, and types of evidence syntheses. All of us must continue to study, teach, and act cooperatively for that to happen.
Michelle Oakman Hayes for her assistance with the graphics, Mike Clarke for his willingness to answer our seemingly arbitrary questions, and Bernard Dan for his encouragement of this project.
All authors participated in the development of the ideas, writing, and review of this manuscript. The author(s) read and approved the final manuscript.
The work of John Ioannidis has been supported by an unrestricted gift from Sue and Bob O’Donnell to Stanford University.
The authors declare no competing interests.
This article has been published simultaneously in BMC Systematic Reviews, Acta Anaesthesiologica Scandinavica, BMC Infectious Diseases, British Journal of Pharmacology, JBI Evidence Synthesis, the Journal of Bone and Joint Surgery Reviews , and the Journal of Pediatric Rehabilitation Medicine .
Publisher’ s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Outcomes of (A) 3-month good functional outcome (modified Rankin scale 0-2), (B) successful recanalization, (C) early neurological improvement, and (D) 3-month excellent functional outcome (modified Rankin Scale 0-1). The size of squares is proportional to the weight of each study. Horizontal lines indicate the 95% CI of each study; diamond, the pooled estimate with 95% CI.
The size of squares is proportional to the weight of each study. Horizontal lines indicate the 95% CI of each study; diamond, the pooled estimate with 95% CI.
eTable 1. Definition of Successful Recanalization in Patients With Confirmed Intracranial Vessel Occlusion Across Included Studies
eTable 2. Definition of Early Neurological Improvement at 24h Across Included Studies
eTable 3. Adjustment for Potential Confounders Across Included Studies
eTable 4. Quality Assessment of Included Studies With the Newcastle-Ottawa Scale
eFigure 1. Unadjusted Analyses on the Comparison Between Tenecteplase and Alteplase for the Outcomes of (A) Symptomatic Intracranial Hemorrhage and (B) Parenchymal Hematoma Following Intravenous Thrombolysis
eFigure 2. Adjusted Analyses on the Comparison Between Tenecteplase and Alteplase for the Outcomes of Symptomatic Intracranial Hemorrhage Following Intravenous Thrombolysis
eFigure 3. Funnel Plots on the Unadjusted and Adjusted Comparison Between Intravenous Tenecteplase and Alteplase for the Primary Outcome of Interest (Three-Month Good Functional Outcome; Modified Rankin Scale 0-2)
Sign up for emails based on your interests, select your interests.
Customize your JAMA Network experience by selecting one or more topics from the list below.
Others also liked.
Katsanos AH , Psychogios K , Turc G, et al. Off-Label Use of Tenecteplase for the Treatment of Acute Ischemic Stroke : A Systematic Review and Meta-analysis . JAMA Netw Open. 2022;5(3):e224506. doi:10.1001/jamanetworkopen.2022.4506
© 2024
Question How does the use of tenecteplase compare with the use of alteplase in the clinical outcomes of patients with acute ischemic stroke (AIS) receiving intravenous thrombolysis?
Findings In this systematic review and meta-analysis, 6 nonrandomized studies including 1820 participants were analyzed. Intravenous tenecteplase was associated with better short-term and long-term functional outcomes in patients with AIS and a higher likelihood of successful recanalization in patients with acute intracranial vessel occlusions; no increased risk of intracranial bleeding was noted with intravenous tenecteplase compared with alteplase.
Meaning Analysis of evidence from nonrandomized studies suggests that tenecteplase is as safe as alteplase for the treatment of AIS and tenecteplase is potentially associated with more favorable outcomes.
Importance Tenecteplase is being evaluated as an alternative thrombolytic agent for the treatment of acute ischemic stroke (AIS) within ongoing randomized clinical trials (RCTs). In addition, nonrandomized clinical experiences with off-label use of tenecteplase vs alteplase for AIS treatment are being published.
Objective To evaluate the available evidence on the safety and efficacy of intravenous tenecteplase compared with intravenous alteplase provided by nonrandomized studies.
Data Sources Eligible studies were identified by searching MEDLINE and Scopus databases. No language or other restrictions were imposed. The literature search was conducted on October 12, 2021. This meta-analysis used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and was written according to the Meta-analysis of Observational Studies in Epidemiology (MOOSE) proposal.
Study Selection Nonrandomized studies (prospective or retrospective) comparing intravenous tenecteplase (at any dose) with intravenous alteplase in patients with AIS were included in the analysis.
Data Extraction and Synthesis The crude odds ratios (ORs) and 95% CIs were calculated for the association of tenecteplase vs alteplase with the outcomes of interest and adjusted ORs were extracted if provided. Estimates using random-effects models were pooled.
Main Outcomes and Measures The primary outcome was the probability of good functional outcome (modified Rankin scale [mRS] score, 0-2) at 90 days.
Results Six studies were identified including a total of 1820 patients (618 [34%] treated with tenecteplase). Patients receiving tenecteplase had higher odds of 3-month good functional outcome (crude odds ratio [OR], 1.22; 95% CI, 0.90-1.66; adjusted OR, 1.60, 95% CI, 1.08-2.37), successful recanalization (crude OR, 2.82; 95% CI, 1.12-7.10; adjusted OR, 2.38; 95% CI, 1.18-4.81), and early neurological improvement (crude OR, 4.88; 95% CI, 2.03-11.71; adjusted OR, 7.60; 95% CI, 1.97-29.41). No significant differences were detected in 3-month excellent functional outcome proportions (mRS score 0-1; crude OR, 1.53; 95% CI, 0.81-2.91; adjusted OR, 2.51; 95% CI, 0.66- 9.49), symptomatic intracranial hemorrhage (crude OR, 0.97; 95% CI, 0.44-2.16; adjusted OR, 1.16; 95% CI, 0.13-10.50), or parenchymal hematoma (crude OR, 1.20; 95% CI, 0.24-5.95).
Conclusions and Relevance Evidence from nonrandomized studies suggests tenecteplase is as safe as alteplase and potentially associated with improved functional outcomes compared with alteplase. Based on these findings, enrollment in the ongoing RCTs appears to be appropriate.
Tenecteplase has a well-characterized mechanism of action with important practical advantages in administration and superior clinical efficacy for patients with large-vessel occlusion, as demonstrated by randomized clinical trials (RCTs). 1 - 3 Despite the advantages of tenecteplase and the recent endorsement of its use in national and international guidelines, 4 - 6 alteplase remains the only regulatory-approved intravenous thrombolytic agent for the treatment of acute ischemic stroke (AIS). Although the use of intravenous tenecteplase for acute stroke treatment is still considered off-label, intravenous tenecteplase is increasingly being used for the treatment of AIS, particularly in countries where tenecteplase has a lower cost than alteplase. 7 - 9
Because several stroke centers around the world have published their local experience with the off-label use of intravenous tenecteplase for AIS, we decided to perform a systematic review and meta-analysis to evaluate the available evidence on the association of intravenous tenecteplase compared with intravenous alteplase with the outcomes provided by these nonrandomized studies.
The systematic review and meta-analysis is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses ( PRISMA ) reporting guideline and adheres to the Meta-analysis of Observational Studies in Epidemiology ( MOOSE ) proposal.
Three of us (A.H.K., K.P., and G. Tsivgoulis) searched MEDLINE and Scopus databases for nonrandomized studies (prospective or retrospective) reporting outcomes of patients with AIS receiving intravenous thrombolysis with either tenecteplase or alteplase at any dose. The last literature search was performed on October 12, 2021. No language or other restrictions were applied in the literature search algorithm. Conference proceedings from the European Stroke Organization, American Stroke Association, and World Stroke Organization were also screened after the database literature search. Studies reporting only experience with intravenous tenecteplase treatment, without including a comparison intravenous alteplase control group, were excluded. In studies with overlapping participant data, we selected a single publication including the highest number of total participants. Case reports and case series were excluded from further consideration. Risk of bias for each eligible study was assessed with the Newcastle-Ottawa Scale by the 2 of us who performed the literature search (A.H.K. and K.P.). 10 This scale uses multiple-choice questions to address the areas of selection, comparability, and exposure/outcome assessment. High-quality ratings are identified with a star and studies can earn a maximum of 9 star-points. 10 All conflicts during the literature search and bias assessment were resolved after discussion.
The primary outcome of interest was a modified Rankin Scale (mRS) score of 0 to 2 at 3 months. 11 Secondary efficacy outcomes of interest included successful recanalization in patients with confirmed intracranial vessel occlusion according to the definition used in each study (eTable 1 in the Supplement ), early neurologic improvement according to the definition used in each study (eTable 2 in the Supplement ), and excellent functional outcome, defined as 3-month mRS scores of 0 or 1. 11 Primary safety outcome included symptomatic intracranial hemorrhage. Any parenchymal hematoma following intravenous thrombolysis treatment constituted the secondary safety end point.
For each outcome of interest, we extracted or calculated the crude odds ratios (ORs) and corresponding 95% CIs. All adjusted ORs and corresponding 95% CIs were extracted from each study. The adjustment for potential confounders in different studies is displayed in eTable 3 in the Supplement . Study estimates were pooled under the random-effects model. Heterogeneity between studies was assessed with the Cochran Q and I 2 statistics. For the qualitative interpretation of heterogeneity, I 2 values of at least 50% were considered to represent substantial heterogeneity, and values of at least 75% indicated considerable heterogeneity. 12 Publication bias was evaluated graphically by inspection of a funnel plot for the primary outcome. 13 The significance threshold was set at α = .05. All statistical analyses were conducted with RevMan, version 5.3 software (Cochrane Collaboration).
We analyzed aggregate data from 6 studies, including 1820 participants (618 [34%] treated with tenecteplase). A literature search retrieved 254 records from MEDLINE and 347 records from Scopus. After excluding duplicates, we identified 8 studies potentially eligible for inclusion. Two of these studies were excluded—one included overlapping participant data with another publication and the other included no intravenous alteplase treatment control group ( Figure 1 ). 14 , 15 Characteristics of the studies that were eligible for the meta-analysis are presented in Table 1 . 16 - 21
In the quality control of included studies, we noted selection issues with cases (tenecteplase treatment) and controls (alteplase treatment) (eTable 4 in the Supplement ). Alemseged et al 16 included only patients with basilar artery occlusion treated with either intravenous tenecteplase or intravenous alteplase, thus limiting the generalizability of their findings. Data in the study by Alemseged et al 16 were derived from a retrospective analysis of patients with basilar artery occlusion prospectively enrolled either in the Tenecteplase vs Alteplase Before Endovascular Therapy for Ischemic Stroke trial or the Basilar Artery Treatment and Management registry. In the study by Parsons et al, 17 controls (alteplase treatment) were patients with AIS presenting within 3 hours from symptom onset and without certain neuroimaging criteria, whereas cases (tenecteplase treatment) represented patients with AIS presenting between 3 and 6 hours from symptom onset who fulfilled certain neuroimaging criteria. In the study by Seners et al, 19 the control group comprised hospitalized patients with AIS who received alteplase treatment at different institutions. In the studies by Psychogios et al 18 and Mahawish et al, 20 data on patients treated with intravenous tenecteplase were prospectively collected; however, those who received intravenous alteplase were from historical cohorts at these same institutions. Regarding outcome assessment, 3 of 6 studies reported no blinding of outcome adjudicators to treatment. 18 - 20 In addition, one of these studies 20 reported a substantial proportion of patients lost to 3-month follow-up with imbalance in lost-to-follow-up rates between patients treated with intravenous alteplase (50 [18%]) or intravenous tenecteplase (77 [14%]).The full text of 1 of the studies had not been published at the time of our systematic review; thus, bias assessment was not possible. 21
Unadjusted and adjusted analyses for primary and secondary outcomes of interest are briefly summarized in Table 2 . Patients receiving tenecteplase had higher odds of 3-month good functional outcome with crude OR (1.22; 95% CI, 0.90-1.66) ( Figure 2 A) and adjusted OR (1.60; 95% CI, 1.08-2.37) ( Figure 3 A), successful recanalization with crude OR (2.82; 95% CI, 1.12-7.10) ( Figure 2 B) and adjusted OR (2.38; 95% CI, 1.18-4.81) ( Figure 3 B), and early neurologic improvement with crude OR (4.88; 95% CI, 2.03-11.71) ( Figure 2 C) and adjusted OR (7.60; 95% CI, 1.97-29.41) ( Figure 3 C). No significant differences were detected in the probability of 3-month excellent functional outcome with crude OR (1.53; 95% CI, 0.81-2.91) ( Figure 2 D) and adjusted OR (2.51, 95% CI, 0.66-9.49) ( Figure 3 D); the risk difference between intravenous tenecteplase and intravenous alteplase in the pooled crude analysis was 10% with the lower bound of the 95% CI (−5% to 26%) for treatment effect difference at −5%. In addition, no significant differences were detected in the probability of 3-month symptomatic intracranial hemorrhage crude OR (0.97; 95% CI, 0.44-2.16) (eFigure 1A in the Supplement ) and adjusted OR (1.16; 95% CI, 0.13-10.50) (eFigure 2 in the Supplement ), or any parenchymal hematoma crude OR (1.20; 95% CI, 0.24-5.95) (eFigure 1B in the Supplement ). There was little heterogeneity in the results provided from included studies, except for the crude association of treatment with successful recanalization ( I 2 = 65%), the adjusted associations of treatment with the likelihood of excellent functional outcome ( I 2 = 54%) and symptomatic intracranial hemorrhage ( I 2 = 69%). Evidence of publication bias (small-study effect) was noted in the funnel plot of the crude (eFigure 3A in the Supplement ) but not of the adjusted association (eFigure 3B in the Supplement ) of treatment with the primary outcome of interest.
To our knowledge, the present study is the first meta-analysis of nonrandomized evidence on the comparison between tenecteplase and alteplase in the treatment of AIS using data from different clinical settings. The results are similar to a meta-analysis of RCT data on this same issue. 22 Tenecteplase use as an intravenous thrombolytic agent for patients with AIS (with and without an underlying large-vessel occlusion) was associated with higher odds of early neurologic improvement and good functional outcome compared with intravenous alteplase. Tenecteplase administration was also associated with a 2-fold higher likelihood of successful recanalization in patients with acute intracranial vessel occlusions compared with intravenous alteplase. No significant difference in the risk of intracranial bleeding between the 2 intravenous thrombolytic agents was noted.
Our findings are in line with evidence from a systematic review and meta-analysis of RCTs, 3 suggesting that patients with confirmed large-vessel occlusions receiving intravenous tenecteplase have a 3-fold higher odds of successful recanalization (OR, 3.05; 95% CI, 1.73-5.40) and 2-fold higher odds of favorable functional outcome (mRS score, 0-2) at 3 months (OR, 2.06; 95% CI, 1.15-3.69), with no significant increase in the risk of intracranial bleeding, compared with those receiving intravenous alteplase. Consistent with our findings, to our knowledge, neither individual RCTs nor other meta-analyses published to date have suggested any safety concerns with the use of tenecteplase compared with alteplase on the risks of intracranial hemorrhage and all-cause mortality. 3 , 22
Tenecteplase seems to achieve fast recanalization in patients with large-vessel occlusion at a consistent rate of 20%, which is independent of whether the patients are secondarily transferred or directly admitted to a center capable of performing endovascular procedures. 23 This analysis also shows that intravenous tenecteplase treatment was associated with an increased likelihood of early neurologic improvement in both crude and adjusted analyses—an association that has been reported in a previous meta-analysis of RCTs. 24 Compelling evidence for tenecteplase superiority as a thrombolytic agent compared with alteplase has been provided to date only for patients with confirmed large-vessel occlusions. 25 , 26 Studies that have included all patients presenting with symptoms suggestive of AIS did not prove the superiority of tenecteplase over alteplase. 27 , 28 This disparity on the outcomes associated with tenecteplase in different study populations has been reflected in the current guidelines from the European Stroke Organization 5 and American Heart Association/American Stroke Association 6 suggesting that tenecteplase treatment may be considered only for patients with confirmed acute large-vessel occlusion who are eligible for both intravenous thrombolysis and subsequent endovascular thrombectomy.
In the present meta-analysis, no difference in the likelihood of 3-month excellent functional outcome was detected between patients receiving intravenous tenecteplase or intravenous alteplase. However, in our analysis, the risk difference between intravenous tenecteplase and intravenous alteplase in the pooled crude analysis was 10% with the lower bound of the 95% CI (−5% to 26%) for treatment effect difference at −5%. This noninferiority margin of −5% for treatment effect difference has been suggested as the minimal clinically important difference for acute stroke therapies by a previous survey of stroke experts and used in a previous meta-analysis comparing intravenous tenecteplase with intravenous alteplase for the treatment of AIS within the setting of RCTs. 22
This study has limitations. First, included studies were nonrandomized; thus, imbalances in patient characteristics are expected between the intravenous tenecteplase and intravenous alteplase groups. For this reason, we also report the adjusted treatment associations with the outcomes of interest. Most of the studies provided adjusted associations for the outcomes of interest using multivariable regression models, with confounders selected either a priori 18 , 20 or from univariable associations. 17 In one study, the process for the selection of covariates included in the multivariable analysis was not provided. 16 Another study used propensity score matching to address imbalances in baseline characteristics, 19 using a priori confounders to estimate the propensity score for each patient (eTable 3 in the Supplement ). Despite the differences in the methods used for adjustment and the selection process for confounders between included studies, no evidence of heterogeneity was evident in the adjusted analyses for the efficacy end points ( Table 2 ). However, the number of studies included in the adjusted analyses is low because some of the eligible studies provided only crude associations for the outcomes of interest. Second, as highlighted in our bias assessment, the selection of the overall population and alteplase control groups in some of the studies raises concerns about unmeasured confounding owing to the different time periods and institutions in which patients in the case and control groups were treated. Third, although in all studies the standard intravenous alteplase dose (0.9 mg/kg) was used, the intravenous tenecteplase dose varied both within and between studies, ranging from 0.1 to 0.4 mg/kg, with most studies using the 0.25-mg/kg dose ( Table 1 ). In patients with AIS due to a large-vessel occlusion, no significant differences in clinical and radiologic end points were documented between the 0.40- and 0.25-mg/kg doses, with the exception of a numerically higher intracranial bleeding risk with the 0.4-mg/kg dose. 29 Fourth, there was no central adjudication of the symptomatic intracranial bleeding and successful recanalization events or blinding of the clinical outcome assessors for the majority of included studies.
This meta-analysis provides supporting evidence from nonrandomized studies that intravenous tenecteplase may be a treatment option for patients with AIS that is associated with more favorable clinical outcomes compared with intravenous alteplase. This hypothesis is being evaluated in ongoing large RCTs examining the utility of intravenous tenecteplase for the treatment of patients with AIS presenting within 4.5 hours 30 - 34 or 24 hours 35 , 36 from symptom onset and in patients presenting after 4.5 hours from stroke onset or with unknown onset time. 37 - 40 Based on the comparable safety profile of both thrombolytic agents, supported by both RCT and nonrandomized evidence, enrollment in the aforementioned ongoing RCTs appears to be appropriate.
Accepted for Publication: February 8, 2022.
Published: March 31, 2022. doi:10.1001/jamanetworkopen.2022.4506
Open Access: This is an open access article distributed under the terms of the CC-BY License . © 2022 Katsanos AH et al. JAMA Network Open .
Corresponding Author: Aristeidis H. Katsanos, MD, Division of Neurology, McMaster University/Population Health Research Institute, 237 Barton St E, Hamilton, ON L8L 2X2, Canada ( [email protected] ).
Author Contributions: Drs Katsanos and Tsivgoulis had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Drs Katsanos and Psychogios contributed equally to the study.
Concept and design: Katsanos, De Marchis, Filippou, Menon, Tsivgoulis.
Acquisition, analysis, or interpretation of data: Katsanos, Psychogios, Turc, Sacco, Aguiar de Sousa, De Marchis, Palaiodimou, Ahmed, Sarraj, Tsivgoulis.
Drafting of the manuscript: Katsanos, Psychogios, Tsivgoulis.
Critical revision of the manuscript for important intellectual content: Psychogios, Turc, Sacco, Aguiar de Sousa, De Marchis, Palaiodimou, Filippou, Ahmed, Sarraj, Menon.
Statistical analysis: Katsanos, Palaiodimou, Tsivgoulis.
Administrative, technical, or material support: Psychogios, De Marchis, Filippou.
Supervision: Aguiar de Sousa, Menon, Tsivgoulis.
Conflict of Interest Disclosures: Dr Katsanos reported participating as an investigator for the Alteplase Compared to Tenecteplase in Patients With Acute Ischemic Stroke trial and being supported by an Internal Career Award from the Department of Medicine, McMaster University. Dr Sacco reported receiving speaker and advisory board participation fees from Allergan, Novartis, Teva, and Eli Lilly and Co; fees for advisory board participation from Lundbeck, AstraZeneca, and Novo-Nordisc for advisory board participation; and speaker fees from Abbott during the conduct of the study. Dr Aguiar de Sousa reported receiving personal fees for AstraZeneca advisory board participation, travel support from Boehringer Ingelheim, and speaking fees from Bayer outside the submitted work. Dr De Marchis reported serving as an unpaid steering committee member of the Tenecteplase in Wake Up Stroke trial. Dr Ahmed reported being chairman of SITS International, which receives a grant from Boehringer Ingelheim for the SITS-International Stroke Thrombolysis Register outside the submitted work. Dr Sarraj reported receiving grants from Stryker Neurovascular and consultant fees from Stryker Neurovascular and AstraZeneca outside the submitted work. Dr Menon reported serving as chief medical officer and director of Circle NVI, holding stock and options, outside the submitted work. No other disclosures were reported.
IMAGES
VIDEO
COMMENTS
A 2012 systematic review of the state of the literature on female Veterans experiencing homelessness found that most of the studies examined epidemiological characteristics and risk factors for homelessness among women Veterans. 13 That review noted that access to care was an area where there was a particular paucity of research, with only two ...
We conducted a systematic review according to PRISMA guidelines [] to identify documents that reported on barriers to data sharing in public health [see Additional file 1].We defined public health data as data that were primarily collected by public health agencies for routine purposes such as disease surveillance or program monitoring without primary intention of research [4, 18-20].
The gap between research and practice or policy is often described as a problem. To identify new barriers of and facilitators to the use of evidence by policymakers, and assess the state of research in this area, we updated a systematic review. Systematic review. We searched online databases including Medline, Embase, SocSci Abstracts, CDS, DARE, Psychlit, Cochrane Library, NHSEED, HTA, PAIS ...
A systematic literature review of patient perspectives of barriers and facilitators to access, adherence, stigma, and persistence to treatment for substance use disorder Explor Res Clin Soc Pharm . 2021 Jun 4:2:100029. doi: 10.1016/j.rcsop.2021.100029.
Topic selection and planning. In recent years, there has been an explosion in the number of systematic reviews conducted and published (Chalmers & Fox 2016, Fontelo & Liu 2018, Page et al 2015) - although a systematic review may be an inappropriate or unnecessary research methodology for answering many research questions.Systematic reviews can be inadvisable for a variety of reasons.
This rapid review is the first step to an in-depth exploration of the barriers to data access, linkage and use; a better understanding of which can aid in creating and implementing effective solutions. ... Pagliari C, Jepson R, Cunningham-Burley S. Public responses to the sharing and linkage of health data for research purposes: A systematic ...
Abstract. Objective: To review original research studies published between 1990 and 2004 on the access and use of medicines in Mexico to assess the knowledge base for reforming Mexico's pharmaceutical policy. Material and methods: A literature review using electronic databases was conducted of original studies published in the last 15 years ...
Autistic individuals report barriers to accessing and receiving healthcare, and experience increased morbidity and mortality. This systematic review synthesizes 31 research studies evaluating interventions implemented to improve the healthcare experiences and/or access of autistic persons. Interventions were most commonly patient-focused (58.1%), focused on supporting the autistic individual ...
Design: A mixed methods systematic review was conducted and includes quantitative and qualitative primary research studies. Data sources and methods: Seven databases were searched for quantitative or qualitative research studies. Studies which addressed access or use of healthcare or social services, which were focused on female Veterans or ...
Literature reviews establish the foundation of academic inquires. However, in the planning field, we lack rigorous systematic reviews. In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in literature ...
A literature search is distinguished from, but integral to, a literature review. Literature reviews are conducted for the purpose of (a) locating information on a topic or identifying gaps in the literature for areas of future study, (b) synthesising conclusions in an area of ambiguity and (c) helping clinicians and researchers inform decision-making and practice guidelines.
With the view to address this challenge, the systematic review method was developed. Systematic reviews aim to inform and facilitate this process through research synthesis of multiple studies, enabling increased and efficient access to evidence.[1,3,4] Systematic reviews and meta-analyses have become increasingly important in healthcare settings.
Similarly, the 2020 review by Eze et al. 9 (which had broader inclusion criteria) reported that 92% of the systematic reviews they included were of a low to critically low quality according to a measurement tool to assess systematic reviews (AMSTAR). In contrast, this current review showed high-quality reporting when articles were assessed ...
This review of barriers to e-book use systematically identifies obstacles to engaging reading experiences. Through the use of an analytical framework, the users being studied, study setting, and methods used in previous work are described in order to identify promising areas for future research. Method.
Risk-based access control model: a systematic literature review (Atlam et al., 2020). A systematic review of the risk-based access control model is provided. According to their search strategy, they chose 44 recent studies to summarize their contributions, analyzed the various risk factors and investigated the used risk estimation techniques.
Aims Over the last two decades, the existence of an open access citation advantage (OACA)—increased citation of articles made available open access (OA)—has been the topic of much discussion. While there has been substantial research to address this question, findings have been contradictory and inconclusive. We conducted a systematic review to compare studies of citations to OA and non-OA ...
Once the identity is established, access to the data is determined by access control policies in a second step. Various policy models with different granularities exist [26].We propose to use ...
A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer. Example: Systematic review. In 2008, Dr. Robert Boyle and his colleagues published a systematic review in ...
The aim of this systematic literature review is to map the longitudinal research in the field of Internet Use (IU) and Problematic Internet Use (PIU) in adolescents and emergent adults. Further, this study endeavours to examine the terminology and instruments utilized in longitudinal IU and PIU research and investigate whether statistically ...
This systematic review and meta-analysis was registered with PROSPERO ... Past research revealed that the quality of vegetarian diets varied between South Asian and American with vegetarians in the US consistently consuming healthier food groups than South Asian vegetarians. 69 Thus, our findings should be cautiously interpreted when applied to ...
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question.
A systematic literature search was conducted in the electronic databases of PubMed, PsychINFO and Google Scholar. The main search terms were "remote neuropsychological intervention or training." The included articles were RCT studies published in English, examining the effectiveness of remote healthcare services in neuropsychological ...
Most research on food insecurity (FI) among college students focuses on inadequate access to affordable, nutritious food on campus. High rates of campus FI (up to 82%) indicate that students struggle to acquire food, maintain well-being, and manage their finances. During college, many students rely on jobs to cover basic living expenses, including food; shortfalls can compromise food access ...
Key Points. Question To what extent has the COVID-19 pandemic been associated with changes in the duration, content, and context of daily screen time among children and adolescents globally?. Findings In this systematic review and meta-analysis of 46 studies including 29 017 youths (≤18 years), pooled estimates comparing estimates taken before and during the COVID-19 pandemic revealed an ...
Notably, Cochrane is the largest single producer of evidence syntheses in biomedical research; however, these only account for 15% of the total [].The World Health Organization requires Cochrane standards be used to develop evidence syntheses that inform their CPGs [].Authors investigating questions of intervention effectiveness in syntheses developed for Cochrane follow the Methodological ...
This study aims to synthesise the findings of research on cloud computing adoption and use in libraries. This systematic literature review is based on Preferred Reporting Items for Systematic Reviews and Meta-Analyses method and comprises publications in the English language, published in the four world-renowned databases.
Key Points. Question How does the use of tenecteplase compare with the use of alteplase in the clinical outcomes of patients with acute ischemic stroke (AIS) receiving intravenous thrombolysis?. Findings In this systematic review and meta-analysis, 6 nonrandomized studies including 1820 participants were analyzed. Intravenous tenecteplase was associated with better short-term and long-term ...
2.3. Data Extraction. We used EndNote X9.3.3 to manage records and filter and exclude duplicates. After article screening through inclusion and exclusion criteria, the following data were extracted from the included articles: first author, year of publication, geographic location, vaccine type, mode of immunization, dose of immunization, region, number of samples, sex, age at the time of ...
Copyediting, typesetting, and review of the resulting proofs will be undertaken on this manuscript before final publication of the Version of Record (VoR). During production and pre-press, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal relate to these versions also.
In today's fast-paced world, getting new employees up to speed through effective and efficient onboarding has become more important than ever. Further accelerated by the COVID-19 pandemic, the use of digital onboarding practices is increasingly widespread in practice. This study presents the results of an interdisciplinary systematic literature review to understand the current state of ...