- University of Texas Libraries
- UT Libraries
- Dell Medical School Library
- Access our Services
- What's New?
- Clinical Practice Guidelines
- Clinical Trials
- Drug Information
- Health and Medical Law
- Point-of-Care Tools
- Test Prep Resources
- Video, Audio, and Images
- Search Tips
- PubMed Guide This link opens in a new window
- Ask the Question
- Acquire the Evidence
- Appraise the Evidence
- Evidence Hierarchy
- EBM Bibliography
- Child Neurology
- Emergency Medicine
- Family Medicine
- Internal Medicine
- Obstetrics & Gynecology
- Orthopaedic Surgery
- Physical Medicine & Rehabilitation
- Mobile Apps
- Citation Managers This link opens in a new window
- Citation Manuals
- General Resources
- Study Types
- Systematic Reviews
- Scoping Reviews
- Rapid Reviews
- Integrative Reviews
- Technical Reports
- Case Reports
- Getting Published
- Selecting a Journal
- Open Access Publishing
- Avoiding Low Quality Open Access
- High Quality Open Access Journals
- Keeping Up with the Literature
- Health Statistics
- Research Funding This link opens in a new window
- Author Metrics
- Article Metrics
- Journal Metrics
- Scholarly Profile Tools
- Health Humanities This link opens in a new window
- Health Equity This link opens in a new window
- UT-Authored Articles
- Resources for DMS COVID-19 Elective
- Types of Literature Reviews
- How to Write a Literature Review
- How to Write the Introduction to a Research Article
- Meeting the review family: exploring review types and associated information retrieval requirements | Health Information and Libraries Journal, 2019
- A typology of reviews: an analysis of 14 review types and associated methodologies | Health Information and Libraries Journal, 2009
- Conceptual recommendations for selecting the most appropriate knowledge synthesis method to answer research questions related to complex evidence | Journal of Clinical Epidemiology, 2016
- Methods for knowledge synthesis: an overview | Heart & Lung: The Journal of Critical Care, 2014
- Not sure what type of review to conduct? Brief descriptions of each type plus tools to help you decide
- Ten simple rules for writing a literature review | PLoS Computational Biology, 2013
- The Purpose, Process, and Methods of Writing a Literature Review | AORN Journal. 2016
- Why, When, Who, What, How, and Where for Trainees Writing Literature Review Articles. | Annals of Biomed Engineering, 2019
- So You Want to Write a Narrative Review Article? | Journal of Cardiothoracic and Anesthesia, 2021
- An Introduction to Writing Narrative and Systematic Reviews - Tasks, Tips and Traps for Aspiring Authors | Heart, Lung, and Circulation, 2018
- The Literature Review: A Foundation for High-Quality Medical Education Research | Journal of Graduate Medical Education, 2016
- Writing an effective literature review : Part I: Mapping the gap | Perspectives on Medical Education, 2018
- Writing an effective literature review : Part II: Citation technique | Perspectives on Medical Education, 2018
- Last Updated: Oct 31, 2023 10:41 AM
- URL: https://guides.lib.utexas.edu/medicine
- USC Libraries
- Research Guides
- Health Sciences
Medical Students Scholarly Project Course
- Literature Review
What is a literature review?
Systematic reviews vs literature reviews, literature reviews - articles, writing literature reviews, frequently used journal article databases.
- Conference Posters This link opens in a new window
- Soft Skills
The literature review is the qualitative summary of evidence on a topic using informal or subjective methods to collect and interpret studies.The literature review can inform a particular research project or can result in a review article publication.
- Aaron L. Writing a literature review article. Radiol Technol. 2008 Nov-Dec; 80(12): 185-6.
- Gasparyan AY, Ayvazyan L, Blackmore H, Kitas GD. Writing a narrative biomedical review: considerations for authors, peer reviewers, and editors. Rheumatol Int. 2011 Nov; 31(11): 1409-17.
- Matharu GS, Buckley CD. Performing a literature review: a necessary skill for any doctor. Student BMJ. 2012; 20:e404. Requires FREE site registration
- Literature Reviews The Writing Center at University of North Carolina at Chapel Hill has created a succinct handout that explains what a literature review is and offer insights into the form and construction of a literature review in the humanities, social sciences, and sciences.
- Review Articles (Health Sciences) Guide Identifies the difference between a systematic review and a literature review. Connects to tools for research, writing, and publishing.
- Systematic Approaches to a Successful Literature Review by Andrew Booth; Diana Papaioannou; Anthea Sutton Call Number: Norris Medical Library, Upper Level, LB 1047.3 B725s 2012
- Documenting your search This resource provides guidance on how to document and save database search strategies.
- Cochrane Database of Systematic Reviews
- Embase This link opens in a new window
- Google Scholar This link opens in a new window
- PsycINFO This link opens in a new window
- Scopus This link opens in a new window
- Web of Science This link opens in a new window
- Next: Data Mgmt >>
- Last Updated: Nov 1, 2023 3:17 PM
- URL: https://libguides.usc.edu/healthsciences/spc
- Open access
- Published: 30 October 2023
Effects of simulation in improving the self-confidence of student nurses in clinical practice: a systematic review
- Nojoud Alrashidi 1 ,
- Eddieson Pasay an 2 ,
- Maha Sanat Alrashedi 1 ,
- Aidah Sanad Alqarni 2 ,
- Ferdinand Gonzales 2 ,
- Enas Mohammed Bassuni 2 ,
- Petelyne Pangket 1 ,
- Lorraine Estadilla 2 ,
- Lizy Sonia Benjamin 2 &
- Kawther Elthayeb Ahmed 2
BMC Medical Education volume 23 , Article number: 815 ( 2023 ) Cite this article
Considering the positive influence of simulation from previous literature may encourage educators to regard it as a valuable teaching strategy in nursing schools. This literature review aims to investigate whether the use of simulation improves undergraduate nurses’ self-confidence during clinical practice.
This study employed a literature-based design. Five academic databases, including CINAHL, EBSCO, ProQuest, PubMed and Medline, were utilised to extract relevant studies using appropriate keywords and Boolean operators. Studies published in the last 15 years (2005–2020) were included in the search. Studies were retrieved using the Exclusion and Inclusion criteria. The Critical Appraisal Skills Programme (CASP) was used to critically appraise the studies.
A total of 15 primary research studies were extracted for review. Four major themes emerged from the review: Improved self-confidence to carry out clinical tasks, Increased ability to work in teams, Improved self-confidence to perform community work and Improved self-confidence in communicating with patients and team members.
Clinical simulation is a useful tool in increasing the self-confidence of nursing students to perform clinical tasks, make clinical judgements, communicate with patients and team members and improve their teamwork. To improve the quality of care received by patients, it is recommended that clinical simulations be integrated into the nursing curriculum in Saudi Arabia. Increasing the confidence of students has been shown to be associated with greater confidence in performing clinical tasks.
Peer Review reports
Nurse educators view repetitive simulation as a beneficial teaching strategy for supporting students’ learning objectives [ 1 ]. It provides an experiential environment that eases the feeling of pressure while on placement [ 2 ]. As an innovative method of teaching using simulation-based clinical education, nursing students are provided the opportunity to hone their clinical and decision-making abilities through a variety of real-life situational encounters without jeopardising the patient’s safety [ 3 ]. Guerrero and colleagues [ 4 ] defined simulation as an educational method for students to gain and enhance professional competence, for example, uses simulation, a technology that simulates real-life scenarios. Simulation is an extremely effective tool for creating a number of different learning outcomes [ 5 ]. Moreover, it could provide an environment where learners can develop skills without jeopardising the safety of others. A previous study maintained that students who participated in simulations showed excellent attitudes towards learning and had increased competence and confidence [ 6 ]. In other words, students finish simulations effectively, they gain confidence in their ability to apply the same abilities in the real world.
In simulation, the concept of self-confidence is the degree of assurance and trust that nursing students have in their capacity to carry out the necessary duties and hone the necessary abilities in a simulated clinical situation. Self-confidence is a key component of nursing education and practice, and it is positively connected with student satisfaction and their capacity for critical thought. Through consistent practice, constructive criticism, and debriefing sessions, self-confidence can be built [ 7 ]. To develop self-confidence in simulation, a safe and encouraging learning environment is essential. For students to take risks and try new things, they must feel comfortable and supported [ 8 ]. In order to do this, the simulation environment needs to foster an atmosphere of mutual respect and cooperation [ 9 ]. When students perform well, instructors can provide them encouragement and good feedback, which can boost their confidence.
In nursing education, simulation is crucial for the formation of clinical judgment [ 10 , 11 ]. It is a cutting-edge method of instruction that aim to replicate key features of clinical cases [ 12 ] with effectiveness and learning impact proven in areas like decision-making and clinical reasoning, simulation is acknowledged as a valuable learning tool in nursing education [ 13 ]. A perfect environment for honing clinical judgment skills is provided by recent advancements in high-fidelity simulation [ 10 ]. Students benefit from simulation by getting hands-on experience, which is crucial for developing clinical judgment, and by getting immediate feedback on how they’re doing [ 10 ]. Even while there is no doubt that clinical judgment is tied to real-world experience, students benefit from clear objectives and honest evaluations of their performance. The use of simulation in professions that involves protecting the lives of people could provide an environment where professionals learn skills without compromising the safety of individuals. Al-Elq [ 14 ] added that, in contrast to learning skills by performing procedures on patients, simulation allows nursing students to acquire clinical skills through practice with a patient model. Simulation tools would serve as an optimal alternative to actual patients in healthcare settings. Overall, the aforementioned suggest that using simulation as a teaching tool is effective in helping nursing students strengthen their clinical judgment. To assist in the development of the knowledge and skills required for successful nursing, educators should think about introducing simulation into their courses.
To support the use of simulation in nursing, a framework for creating and executing simulation-based education programs that are efficient, successful, and evidence-based is provided by the NLN Jeffries Simulation Theory [ 15 ]. The idea places a strong emphasis on the value of context, background, design, instructional practices, simulation experience, and outcomes in simulation-based learning. Indeed, it is an effective tool for researchers and educators who are interested in developing and examining simulation-based nursing education programs. Simulation is an evidence-based, learner-centered, collaborative learning and improvement technique [ 16 ]. The best available evidence should be used to inform the design and implementation of simulation in order to fulfil the demands of the students. Building teamwork and communication skills, improving patient outcomes, and enhancing learning are all possible using simulation.
While there is a paucity of literature on the use of simulation in nursing curricula in Saudi Arabia, consideration should also be made that simulation is an emerging tool in nursing education in Saudi Arabia; hence, there is a lack of studies investigating the influence of this type of teaching method on student learning in the country. However, this dearth of literature does not imply that nursing curriculum developers overlooked the potential of simulation as a means of improving clinical competence, teamwork and other needed skills. Notably, this review is important because it assists nursing curriculum developers in making decisions on the use and application of simulation in nursing education. Determining whether the impact of simulation extends beyond the student years would help improve the current knowledge of its effectiveness. Analysing and evaluating studies helps nursing educators take advantage of the potential uses of simulation in developing nurses who are clinically competent. Consequently, this study aims to find evidence from the literature on whether the use of simulation improves undergraduate nurses’ self-confidence in performing their clinical tasks. The goal of this review was to answer the research question—Does simulation increase the self-confidence of student nurses? This can be translated into PICO as: P: Student nurses; I: Simulation; C: No simulation; O: Increase in self-confidence.
Materials and methods
Academic databases, including CINAHL, PubMed, Ebsco and ProQuest, were used to identify and access recent and relevant published studies on the use of simulation amongst nursing students and whether simulation could increase the confidence of undergraduate nursing students during their clinical practice. The Boolean operator AND was used to identify studies that contained the keywords of the search terms. This was to ensure that the retrieved studies would contribute to new knowledge or validate what has been previously known about clinical simulation. The keywords entered into these databases included simulation AND nursing students AND confidence, simulation AND nursing AND confidence, nursing AND simulation, and simulation AND confidence. The Boolean operator ‘AND’ ensured that all keywords entered in the database would be reflected in the studies retrieved for the review.
Inclusion and exclusion criteria
The inclusion criteria of this systematic review includes (1) Studies published between 2005 and 2020 because the use of clinical simulations in the nursing curriculum have increased in the last 15 years, (2) studies published in English, (3) studies that are in full text, and (4) primary research studies to ensure that first-hand evidence was analysed and applied to nursing curriculum development. Exclusion criteria are as follows (1) studies that are not peer-reviewed, and (2) studies that do not measure the outcomes of interest.
The studies were critiqued using the Critical Appraisal Skills Programme (CASP) [ 17 , 18 ] tools for critiquing quantitative and qualitative studies. These tools were chosen because of their credibility in critiquing studies. The critiquing framework for quantitative studies developed by Long et al. [ 12 ] was used to determine the quality of the quasi-experimental and survey studies included in the present review. The CASP tool was also used to examine whether the results could be applied to the local population and whether all clinically important outcomes were taken into consideration.
The CASP [ 17 ] tool for qualitative studies was also used. It begins with three screening questions that review the results, their validity and whether those results can be applied to local populations. This tool contains 10 questions. It starts with a question on whether the aims and objectives are clearly stated and investigates whether a qualitative methodology was appropriate to answer the aims and objectives. Then, it examines the appropriateness of the research design to address the research aims and objectives.
A total of 15 studies that met the inclusion criteria set in this study were retrieved from academic databases. A detailed review of the studies is provided in Table 1 . We used three phases in the conduct of this review.
In the first phase, we identify all of the relevant literature on the topic of interest. This is done by searching electronic databases and other sources of information. The search results are then screened to identify the articles that are most relevant to the review question. This screening process is typically done by reviewing the titles and abstracts of the articles. The second phase is the eligibility assessment where once the relevant articles have been identified, we assessed for their eligibility. This involves reviewing the full text of each article to determine whether it meets the inclusion and exclusion criteria of the review. The inclusion criteria are the characteristics that the articles must have in order to be included in the review. The exclusion criteria are the characteristics that disqualify an article from being included in the review. The third phase is the data extraction and synthesis. Once the eligible articles have been identified, the data is extracted from each article. This data is then synthesized to provide a comprehensive overview of the evidence on the topic of interest. The synthesis involved summarizing the findings of the individual studies, comparing the findings of different studies, and identifying any patterns or trends in the data.
Duplicates were eliminated and discrepancies were resolved by consensus to address issues of rigor and bias and this ensures that each article is only counted once. We did not register our review in the International Prospective Register of Systematic Reviews (PROSPERO) because it did not meet the qualifying requirements.
Of the 15 studies, eight were conducted using quasi-experimental designs [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ]. The participants in the studies are nursing students, nurses, and other healthcare professionals where the interventions in the studies all involve clinical simulation. While the outcomes of the studies vary, they all suggest that clinical simulation can have a positive impact on healthcare professionals’ knowledge, skills, and confidence and provided strong evidence that clinical simulation is an effective teaching and training method for healthcare professionals. In addition, three studies [ 28 , 29 , 30 ] were conducted using randomized control where they have provided strong evidence that clinical simulation is an effective teaching and training method for healthcare professionals. Four studies conducted employing descriptive correlation and evaluation analysis [ 31 , 32 , 33 ] also provide strong evidence that simulation is an effective teaching and training method for healthcare professionals. These studies also provide important information about the factors that contribute to student satisfaction and confidence in simulation training.
The succeeding figure summarises the search and retrieval of the 15 studies (See Fig. 1 ). These findings are arranged according to the main themes that were common amongst them. Four major themes emerged in the review: Improved self-confidence to carry out clinical tasks, Increased ability to work in teams, Improved self-confidence to perform community work and Improved self-confidence in communicating with patients and team members. Each of these themes will be discussed along with the studies that support them.
Retrieval of studies from academic databases
Thematic analysis is an identification of the major themes emerging from the critical appraisal and review of individual studies [ 17 ]. The succeeding section presents the four major themes that emerged in the study. It also looks at the studies that support each of these themes. Figure 2 presents the integrated main themes of the review.
Integrated main themes
Theme 1: improved self-confidence in carrying out clinical tasks
Most of the studies [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 ] suggested that clinical simulation improves the self-confidence of student nurses in carrying out clinical tasks. They found self-confidence to be associated with nurses making more accurate clinical judgements during critical situations. While experienced registered nurses were found to be overconfident, it was suggested that student nurses were under confident in making clinical judgements. However, when student nurses received clinical simulation training, their level of self-confidence improved. This is essential not only in making clinical judgements, but also when performing important procedures for patients. Since these nurses have practised virtually or on patient models, they perceive that they can perform similar procedures with real patients.
Clinical simulation provides a relatively safe environment, since student nurses initially perform procedures on patient models and not on live patients. This provides them with time to think through a task and perform procedures they have learned in the classroom and during simulations. Since they receive feedback if they perform tasks erroneously, they immediately become aware of how to perform the procedure correctly. This could provide them with the needed self-confidence and assurance that they can perform correct procedures on real patients.
Theme 2: increased self-confidence to work in teams
Some of the studies [ 22 , 26 , 29 ] in this review supported the theme that engagement in clinical simulation training increases the self-confidence of student nurses to work in teams. This theme is also related to the first theme, in which practicing their skills on virtual patients helped improve their confidence in performing the same procedures on real patients. Since most nurses are involved in teamwork, the knowledge and skills they gained in performing tasks could have improved their self-confidence to share their learning with other team members. Since effective teamwork is essential in improving the health outcomes of patients, this theme has important implications for nursing practice. The Ministry of Health in Saudi Arabia strongly supports teamwork between nurses and other healthcare practitioners. Developing effective teamwork skills early in the nursing undergraduate years will prepare student nurses to be effective team members in their post-registration years. Nurse educators should consider the integration of this tool in the nursing curriculum in Saudi Arabia because clinical simulation is associated with increased confidence in participating in teams.
Theme 3: improved self-confidence in performing community work
Three of the studies [ 22 , 25 , 33 ] suggested that clinical simulation allowed student nurses to gain the self-confidence to perform community work. These findings have important implications in nursing practice, since the work of nurses is not limited to clinical settings and it involves community work as well. Increasing the self-confidence of nurses to perform community work could help improve the quality of care. Moreover, these nurses are now empowered to increase the self-efficacy of patients during their community work.
Theme 4: improved self-confidence in communicating with patients and team members
Five studies [ 19 , 22 , 26 , 29 , 32 ] reflected the theme that attendance in clinical simulations is associated with improved self-confidence in communicating with patients and team members. Communication [ 34 ] plays an integral role in improving the quality of care received by patients. Because team members are able to articulate their feelings, perceptions and ideas with the team, positive communication influences the quality of the healthcare that they provide. Gaining self-confidence to contribute to the team’s goals and objectives would help nurses set goals that could help improve the quality of life of patients. In the present review, it is suggested that participation in clinical simulation classes will allow student nurses to gain the confidence to communicate with patients and other team members. When sustained over time, this confidence will allow student nurses to develop their communication skills and interact effectively with patients and colleagues.
This review aims to find evidence from the literature on whether the use of simulation improves undergraduate nurses’ self-confidence in performing their clinical tasks. The primary goal of simulation training is to ensure that student nurses acquire the requisite skills to perform clinical procedures, participate in collaborative work with other healthcare practitioners and enhance safety during the actual care of patients [ 14 ]. One way to achieve quality care is to ensure that student and registered nurses are confident in carrying out clinical tasks and procedures and in working effectively with patients and their family members in community and other healthcare settings. The capability of student nurses to be confident in making clinical judgements and performing clinical procedures is a critical part of effective and safe healthcare [ 34 ]. The results of the present review strongly suggest that attendance in simulation training improves the ability of student nurses to conduct Advanced Cardiovascular Life Support (ACLS) and perform procedures such as peripheral venous catheterisation. Both procedures require repeated practice before student nurses are regarded as competent by their mentors [ 34 ]. Apart from refining the skills of the student nurses, simulation training additionally improved the teamwork of the students with other healthcare professionals. This is an important trait since nurses are encouraged to lead healthcare and collaborate with other healthcare practitioners when creating healthcare innovations or planning the care and management of the healthcare conditions of patients [ 35 ].
Developing the self-confidence of student nurses at an early stage might improve their ability to provide safe and effective healthcare. Most of the studies included in this review showed a strong association between simulation training and an increase in the reported self-confidence of student nurses. Under confidence to perform clinical tasks or make clinical judgements could compromise patient safety [ 35 ]. Gao et al. [ 36 ] argued that both overconfidence and under confidence in making judgements during critical events may result in inappropriate or delayed interventions. Nurses might experience being over or under confident in making decisions. Gao et al. [ 36 ] also reiterate that the overconfidence of nurses in making decisions makes them prone to making incorrect decisions because they are less likely to actively seek the help of others. This increases their likelihood of making erroneous clinical judgements, and it compromises patient safety. Similarly, nurses being under confident might lead to delays in making clinical judgments or interventions.
Increasing the confidence of student nurses is crucial to helping them make appropriate clinical judgements and perform clinical procedures. Goldhill [ 37 ] suggested that confidence in making clinical judgements and performing procedures appears to be linked to experience and difficulty in making judgements. The findings of Yang et al. [ 24 ] maintained that when compared to registered nurses, student nurses tended to report being under confident when making clinical judgements, even following simulation training. This contrasts with the rest of the studies, which argued that student nurses were generally confident in performing clinical procedures, communicating interprofessionally and participating in teamwork. However, Yang et al. [ 24 ] compared the findings of student nurses with those of registered nurses. This marked difference could have contributed to the observation that student nurses were generally less confident in making clinical judgements.
These findings have an important connection with the results of the other studies reviewed in the present study. It should be noted that all the interventions were only done for a few sessions, with one session only lasting three hours [ 22 ]. This promotes safe and effective care and ensures that students learn in a safe environment without the risk of harming patients [ 35 ]. This also suggests that, even with limited exposure time to simulation training, this significantly increases the self-confidence of student nurses. In addition to clinical simulation, the continuous experiences of the nurses as they progress from being students to registered nurses would add to their confidence in clinical judgement and procedures. The focus of this review is limited to investigating the relationship between clinical simulation and nurses’ confidence in performing procedures or clinical tasks. However, Yang et al. [ 24 ] suggested that confidence in clinical judgement might be developed after several experiences with an acute or critical event and not exclusively through simulations in classroom settings.
Crookes et al. [ 38 ] stated that an appropriate level of confidence reflects an individual’s clinical experience, and it also marks one’s competency. Therefore, clinical experience is suggested to be a factor in developing the confidence of nurses when making clinical judgements. The present review shows that even amongst student nurses, confidence in performing clinical tasks and making clinical decisions could be increased after simulation training. Nonetheless, the findings of this review should still be taken with caution. All the studies had a relatively small sample size, which limits their application to a larger and more heterogeneous population [ 39 ]. The outcomes measured in the studies were also different. However, all the studies showed that the self-confidence to perform clinical tasks, make clinical judgements, participate in teams and communicate with patients was evident amongst the student nurses following attendance in clinical simulation sessions. It is noteworthy that although most of these sessions only lasted for a few hours, they had an impact on the knowledge and self-confidence of the participants. This suggests that clinical simulation could be considered an important tool in helping student nurses gain self-confidence.
Hart et al. [ 26 ] explained that it is essential for nurse educators to use innovations during teaching to address the gap between theory and practice in nursing. The use of innovations in clinical simulation in teaching students how to assess a patient, communicate with them, perform clinical procedures and make clinical judgements has already been used for several years in numerous countries throughout the world. However, its application in Saudi Arabia has yet to be maximised, especially in the nursing curriculum. As shown in the studies reviewed for this investigation, there is strong evidence that clinical simulation is effective in helping student nurses gain confidence. Since all the studies included student nurses as part of the sample population, it should be noted that this self-confidence was further enhanced during post-registration [ 35 ]. Intensive care nurses, for example, expressed high satisfaction with simulation-based learning. Further, it also enhanced their self-confidence in performing critical procedures for their patients.
Students learn differently, and their prior learning could influence how they accept clinical simulation as a teaching strategy. In Saudi Arabia, for example, student nurses learn through theory and practice. The latter is accomplished during placements in different clinical settings and is supervised by senior nurses when performing clinical tasks [ 9 ], which provides them an opportunity to enhance their clinical skills. However, this also increases the risk of compromising patient safety, especially if students are not confident in performing clinical tasks [ 35 ]. At present, there are no established guidelines on the number of hours that could be used for clinical simulation in Saudi Arabia’s undergraduate nursing curriculum. This contrasts with the UK [ 40 ], where educators are allowed to incorporate 300 h of simulation as part of teaching actual clinical skills to nursing students.
The present literature-based study could provide a background on how clinical simulation improves patient safety by increasing the self-confidence of student nurses. This self-confidence could extend until post-registration, and it would enable the student nurses to be better prepared in their future roles. The wide application of clinical simulation in clinical practice may also promote teamwork. Nurses frequently work in teams, and developing them to be confident team contributors ensures that they become active participants during multidisciplinary teamwork [ 35 ].
As previously noted, each student learns differently. Introducing a simulation-based curriculum should be done only after students have sufficient knowledge and experience to provide meaning to the instructions given during simulation. Jarvis and Rivers [ 41 ] utilised constructivist theory as the basis for developing critical thinking skills amongst students who were enrolled in a simulation-based curriculum. Learners are identified as visual, audio read/write and kinaesthetic learners (VARK) [ 34 ]. Clinical simulation could benefit kinaesthetic learners the most since it allows them to practise their skills on virtual or model patients. Visual and audio learners could benefit from clinical simulation since they can hear and see clinical procedures. In contrast, read/write learners may benefit from this type of learning as they reflect on the learning and write down their reflections or read about the clinical task before their participation in clinical simulation training or sessions. The findings of the studies also suggested the impact of clinical simulation on the future practice of nursing students. As students gain self-confidence, their ability to provide quality care also improves.
The introduction of a simulation-based nursing curriculum in Saudi Arabia was preceded by the introduction of a similar curriculum in the medical field [ 14 ]. Integrating simulation-based learning in the nursing curriculum in Saudi Arabia might increase the self-confidence of nursing students. Notably, the present review showed that simulation is strongly associated with improvements in self-confidence when communicating with patients and colleagues. McCabe [ 42 ] argued that effective communication is possible when nurses have the confidence to communicate effectively with patients. Increasing the competence of nurses has been noted as an effective means of increasing their self-confidence [ 42 ]. As demonstrated in this review, the introduction of clinical simulation and the actual practice of students on what they learned on model or virtual patients could have resulted in perceived competence in clinical skills. This could have allowed nurse students to perceive that they are not only confident in their clinical skills, but also in communicating what they have learned with patients and their colleagues. The studies included in this review strongly suggested that the quality of care is enhanced when the nurse-patient relationship is enhanced. Since effective communication underpins the nurse-patient relationship, increasing the self-confidence of nurses to communicate with their patients is one way of improving patient care.
Limitations of the study
Despite limitations, it is noteworthy that all the studies were able to demonstrate that, even with short training sessions on clinical simulation, its impact on the self-reported self-confidence of the students was significant. There were noteworthy positive changes following the learning intervention. Although most of these significant changes were seen in pre-and post-test evaluations, and not through comparisons with a control group, the findings were able to establish that clinical simulations could improve the self-confidence of the student nurses. Conversely, we did not include grey literature, which may have had a possible benefit on the results. Also, the majority of the studies originated from Western countries; thus, extrapolation of these results to Eastern populations is questionable. Meanwhile, many of the studies suffered from significant sources of bias, which should be taken into consideration in future interrogations. On many occasions, the effect was assessed by very few studies; thus, the evidence to support it was low, and the small sample sizes further acted as a limitation on the representativeness of the findings to the population of student nurses.
Implications to nursing/medical education and practice
This review has significant ramifications for nursing and medical practice. The self-confidence of students needs to grow if they are to develop post-registration skills and be confident in handling their duties. As this review has demonstrated, developing competence in carrying out clinical tasks or procedures is related to one’s level of confidence. According to research, clinical simulation training can assist students in becoming proficient in their chosen clinical procedures, clinical judgement and interpersonal communication. The students’ performance in class and the standard of care given to patients may increase once they believe that they are now capable of communicating and carrying out activities. Finding solutions to improve quality care is crucial for nurse educators because the goal of healthcare is to always deliver patient-centred care.
According to this review, clinical simulation is a technique that can both boost students’ self-confidence and enhance the standard of patient care. It is noteworthy that there is little current research exploring how well clinical simulation can boost students’ self-confidence in Saudi Arabia. The results of this review may be used by educators, legislators and other interested parties to assess the relationship between clinical simulation and student confidence growth. The results of this review could also be applied to improve patients’ communication abilities. The ability to communicate with patients more confidently has been linked to clinical simulations. Students get a better understanding of patients’ needs once they feel comfortable speaking with them.
Conclusion and recommendations
A total of 15 studies were retrieved for this literature-based study. The relevance of these studies ensured that the latest evidence on the use of clinical simulation in improving the self-confidence of student nurses was evaluated. Appropriate critiquing tools were utilised to evaluate each study. The findings of this review suggest that the self-confidence of student nurses is significantly improved following attendance in clinical simulation training in their undergraduate nursing curriculum. However, the research designs utilised in the studies included in this review act as a limitation to the applicability of the findings to a larger and more heterogeneous group.
Contribution to the field statement
The inclusion of simulation-based learning in Saudi Arabia’s nursing curriculum may boost nursing students’ self-confidence. The present review, which is significant, demonstrated that simulation is substantially linked to improvements in self-confidence when speaking with patients and coworkers.
Availability of data and materials
The raw data supporting the conclusions of this article will be made available to the corresponding author (Nojoud Alrashidi) without undue reservation by the authors.
Al Gharibi KA, Arulappan J. Repeated simulation experience on self-confidence, critical thinking, and competence of nurses and nursing students—An integrative review. SAGE Open Nurs. 2020;6:237796082092737. https://doi.org/10.1177/2377960820927377 . (SAGE Publications).
Article Google Scholar
Hutchinson TL, Janiszewski Goodin H. Nursing student anxiety as a context for teaching/learning. J Holist Nurs. 2012;31(1):19–24. https://doi.org/10.1177/0898010112462067 . (SAGE Publications).
Kim J, Park JH, Shin S. Effectiveness of simulation-based nursing education depending on fidelity: a meta-analysis. BMC Med Educ. 2016;16(1). https://doi.org/10.1186/s12909-016-0672-7 . Springer Science and Business Media LLC.
Guerrero JG, Tungpalan-Castro GM, Al Fergani B, Gomma NH, Hafiz AH, Pingue-Raguini M. Impact of high-fidelity and virtual simulation experiences on nurses’ acquired knowledge and skills for triaging suspected COVID-19 patients. Comput Educ. 2022;1100002. https://doi.org/10.1016/j.cexr.2022.100002 .
Sierra J. The potential of simulations for developing multiple learning outcomes: the student perspective. Int J Manag Educ. 2020;18(1): 100361. https://doi.org/10.1016/j.ijme.2019.100361 .
Dearmon V, Graves RJ, Hayden S, Mulekar MS, Lawrence SM, Jones L, Smith KK, Farmer JE. Effectiveness of simulation-based orientation of baccalaureate nursing students preparing for their first clinical experience. J Nurs Educ. 2013;52(1):29–38. https://doi.org/10.3928/01484834-20121212-02 . (Available from:).
Perry P. Concept analysis: confidence/self-confidence. Nurs Forum. 2011;46(4):218–30. https://doi.org/10.1111/j.1744-6198.2011.00230.x .
Koukourikos K, Tsaloglidou A, Kourkouta L, Papathanasiou IV, Iliadis C, Fratzana A, Panagiotou A. Simulation in clinical nursing education. Acta Inform Med. 2021;29(1):15–20. https://doi.org/10.5455/aim.2021.29.15-20 .
Lateef F. Maximizing learning and creativity: understanding psychological safety in simulation-based learning. J Emerg Trauma Shock. 2020;13(1):5–14. https://doi.org/104103/JETSJETS_96_19.
Lasater K. Clinical judgment development: using simulation to create an assessment rubric. J Nurs Educ. 2007;46(11):496–503. https://doi.org/10.3928/01484834-20071101-04.
Rishcer KD. Developing clinical judgment in simulation. Preparing nurses for practice. 2021. From https://www.keithrn.com/2021/09/developing-clinical-judgment-in-simulation/ .
Yang F, Wang Y, Yang C, Zhou MH, Shu J, Fu B, Hu H. Improving clinical judgment by simulation: a randomized trial and validation of the Lasater clinical judgment rubric in Chinese. BMC Med Educ. 2019;19(1):20.
Theobald KA, Tutticci N, Ramsbotham J, Johnston S. Effectiveness of using simulation in the development of clinical reasoning in undergraduate nursing students: a systematic review. Nurse Educ Pract. 2021;57: 103220. https://doi.org/10.1016/j.nepr.2021.103220 .
Al-Elq AH. Simulation-based medical teaching and learning. J Family Community Med. 2010;17(1):35–40. https://doi.org/10.4103/1319-1683.68787 .
Jeffries PR, Rodgers B, Adamson K. NLN Jeffries simulation theory: brief narrative description. Nurs Educ Perspect. 2015;36(5):292–3. https://doi.org/10.5480/1536-5026-36.5.292 .
Salifu DA, Christmals CD, Reitsma GM. Frameworks for the design, implementation, and evaluation of simulation-based nursing education: a scoping review. Nurs Health Sci. 2022;24(3):545–63. https://doi.org/10.1111/nhs.12955 .
Critical Appraisal Skills Programme: 11 questions to help you make sense of a trial England. CASP, c2013a . Available from: http://www.casp-uk.net/#!casp-tools-checklists/c18f8 . Cited 24 June 2023.
Critical Appraisal Skills Programme: 10 questions to help you make sense of qualitative research England. CASP, c2013b. Available from: http://www.casp-uk.net/#!casp-tools-checklists/c18f8 . Cited 24 June 2023.
Bambini D, Washburn J, Perkins R. Outcomes of clinical simulation for novice nursing students: communication, confidence, clinical judgment. Nurs Educ Perspect. 2009;30(2):79–82.
Blum CA, Borglund S, Parcells D. High-fidelity nursing simulation: Impact on student self-confidence and clinical competence. Int J Nurs Educ Scholarsh. 2010;7:Article 18. https://doi.org/10.2202/1548-923X.2035 .
Thomas C, Mackey E. Influence of a clinical simulation elective on baccalaureate nursing student clinical confidence. J Nurs Educ. 2012;51(4):236–9. https://doi.org/10.3928/01484834-20120224-03 .
Liaw SY, Zhou WT, Lau TC, Siau C, Chan SW. An interprofessional communication training using simulation to enhance safe care for a deteriorating patient. Nurse Educ Today. 2014;34(2):259–64. https://doi.org/10.1016/j.nedt.2013.02.019 . (Elsevier BV).
Mager DR, Campbell SH. Home care simulation for student nurses: medication management in the home. Nurse Educ Today. 2013;33(11):1416–21. https://doi.org/10.1016/j.nedt.2012.11.007 .
Yang H, Thompson C, Bland M. The effect of clinical experience, judgment task difficulty and time pressure on nurses’ confidence calibration in a high fidelity clinical simulation. BMC Med Inform Decis Mak. 2012;12(1). https://doi.org/10.1186/1472-6947-12-113 . Springer Science and Business Media LLC.
Richards E, Simpson V, Aaltonen P, Krebs L, Davis L. Public health nursing student home visit preparation. Home Healthc Nurse. 2010;28(10):631–8. https://doi.org/10.1097/nhh.0b013e3181f85e10. (Ovid Technologies (Wolters Kluwer Health)).
Hart PL, Brannan JD, Long JM, Maguire MB, Brooks BK, Robley LR. Effectiveness of a structured curriculum focused on recognition and response to acute patient deterioration in an undergraduate BSN program. Nurse Educ Pract. 2014;14(1):30–6. https://doi.org/10.1016/j.nepr.2013.06.010 .
Tawalbeh LI, Tubaishat A. Effect of simulation on knowledge of advanced cardiac life support, knowledge retention, and confidence of nursing students in Jordan. J Nurs Educ. 2014;53(1):38–44. https://doi.org/10.3928/01484834-20131218-01 . (SLACK, Inc.).
Valizadeh L, Amini A, Fathi-Azar E, Ghiasvandian S, Akbarzadeh B. The effect of simulation teaching on baccalaureate nursing students’ self-confidence related to peripheral venous catheterization in children: a randomized trial. J Caring Sci. 2013;2(2):157–64. https://doi.org/10.5681/jcs.2013.019 .
Clay-Williams R, McIntosh CA, Kerridge R, Braithwaite J. Classroom and simulation team training: a randomized controlled trial. Int J Qual Health Care. 2013;25(3):314–21. https://doi.org/10.1093/intqhc/mzt027 . (Available from:).
Smith SJ, Roehrs CJ. High-fidelity simulation: factors correlated with nursing student satisfaction and self-confidence. Nurs Educ Perspect. 2009;30(2):74–8.
Franklin AE, Burns P, Lee CS. Psychometric testing on the NLN student satisfaction and self-confidence in learning, simulation design scale, and educational practices questionnaire using a sample of pre-licensure novice nurses. Nurse Educ Today. 2014;34(10):1298–304. https://doi.org/10.1016/j.nedt.2014.06.011 . (Available from:).
Ballangrud R, Hall-Lord ML, Hedelin B, Persenius M. Intensive care unit nurses’ evaluation of simulation used for team training. Nurs Crit Care. 2014;19(4):175–84. https://doi.org/10.1111/nicc.12031 .
Kjellin A, Hedman L, Escher C, Felländer-Tsai L. Hybrid simulation: bringing motivation to the art of teamwork training in the operating room. Scand J Surg. 2014;103(4):232–6. https://doi.org/10.1177/1457496913516897 . (SAGE Publications).
Bastable SB. Nurse as educator: principles of teaching and learning for nursing practice. Massachusetts (USA): Jones & Bartlett Learning; 2014. p. 715.
Overstreet M. The use of simulation technology in the education of nursing students. Nurs Clin North Am. 2008;43(4):593–603. https://doi.org/10.1016/j.cnur.2008.06.009 . (Elsevier BV).
Gao H, McDonnell A, Harrison DA, Moore T, Adam S, Daly K, Esmonde L, Goldhill DR, Parry GJ, Rashidian A, Subbe CP, Harvey S. Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med. 2007;33(4):667–79. https://doi.org/10.1007/s00134-007-0532-3 . (Available from:).
Goldhill DR. The critically ill: following your MEWS. QJM. 2001;94(10):507–10. https://doi.org/10.1093/qjmed/94.10.507 . (Available from:).
Crookes PA, Davies S. Research into practice: essential skills for interpreting and applying research in nursing and health care. 2nd ed. Edinburgh (UK): Bailliere Tindall Publishers; 2004. p. 248.
Polit D, Beck C, Hungler B. Essentials of nursing research, methods, appraisal and utilization. 8th ed. Philadelphia: Lippincott Williams & Wilkins; 2013. p. 245.
Nursing and Midwifery Council (NMC) c2010. Available from: https://www.nmc.org.uk/globalassets/sitedocuments/standards/nmc-standards-for-pre-registration-nursing-education.pdf . Cited 24 June 2023.
Jarvis M, Rivers J. Campus clinical. Nurse Educ. 2014;39(4):179–83. https://doi.org/10.1097/nne.0000000000000045 . (Ovid Technologies (Wolters Kluwer Health)).
McCabe C. Nurse-patient communication: an exploration of patients’ experiences. J Clin Nurs. 2004;13(1):41–9. https://doi.org/10.1111/j.1365-2702.2004.00817.x .
We would like to acknowledge the Deanship of Scientific Research at King Khalid University for funding this work through the Small-Group Research Project under grant number RGP1/406/44.
The authors extend their appreciation to the Deanship of Scientific Research at King Khalid University for funding this work through the Small-Group Research Project under grant number RGP1/406/44.
Authors and affiliations.
College of Nursing, University of Hail, Hail City, Saudi Arabia
Nojoud Alrashidi, Maha Sanat Alrashedi & Petelyne Pangket
College of Nursing, King Khalid University, Abha, Saudi Arabia
Eddieson Pasay an, Aidah Sanad Alqarni, Ferdinand Gonzales, Enas Mohammed Bassuni, Lorraine Estadilla, Lizy Sonia Benjamin & Kawther Elthayeb Ahmed
You can also search for this author in PubMed Google Scholar
N.A.A, E.A.P and M.S.A were involved in the conception and design of the study. A.S.A, F.G and E.B organised the database. N.A.A, M.S.A and E.A.P wrote the first draft of the manuscript. A.S.A, L.E, P.P. and E.B and LSB, K.A wrote sections of the manuscript. All authors contributed to manuscript revision and read and approved the submitted version.
Correspondence to Nojoud Alrashidi .
Ethics approval and consent to participate.
Consent for publication
The authors declare no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Reprints and Permissions
About this article
Cite this article.
Alrashidi, N., Pasay an, E., Alrashedi, M.S. et al. Effects of simulation in improving the self-confidence of student nurses in clinical practice: a systematic review. BMC Med Educ 23 , 815 (2023). https://doi.org/10.1186/s12909-023-04793-1
Received : 13 July 2023
Accepted : 20 October 2023
Published : 30 October 2023
DOI : https://doi.org/10.1186/s12909-023-04793-1
Share this article
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative
- Student nurses
- Clinical practice
BMC Medical Education
Neurosyphilis in the modern era: Literature review and case series
- 1 Lomonosov Moscow State University, Faculty of Fundamental Medicine, Moscow, Russia; Bujanov Moscow City Clinical Hospital, Moscow, Russia.
- 2 Institute of Higher Nervous Activity and Neurophysiology, Moscow, Russia; Bujanov Moscow City Clinical Hospital, Moscow, Russia. Electronic address: [email protected].
- 3 Bujanov Moscow City Clinical Hospital, Moscow, Russia.
- 4 Moscow Scientific and Practical Center for Dermatology, Venerology and Cosmetology, Moscow Healthcare Department, Russia.
- 5 Institute of Higher Nervous Activity and Neurophysiology, Moscow, Russia; Bujanov Moscow City Clinical Hospital, Moscow, Russia; Moscow Research and Clinical Center for Neuropsychiatry, Moscow Healthcare Department, Russia.
- 6 Moscow Research and Clinical Center for Neuropsychiatry, Moscow Healthcare Department, Russia; Pirogov Russian National Research Medical University, Moscow, Russia.
- PMID: 31427238
- DOI: 10.1016/j.jocn.2019.08.033
The term of neurosyphilis (NS) refers to infection of central nervous system by Treponema pallidum. Classically, it has been divided into early (meningitis, meningovascular) and late forms (general paresis and tabes dorsalis). The availability of penicillin and high sensitivity of Treponema pallidum to this antibiotic has led to a widely held perception about rarity of syphilitic forms with central nervous system involvement. However, patient can exhibit atypical clinical presentation. Recently different clinical cases with autoimmune encephalitis-mimicking presentation or atypical movement disorders were described. In this article we presented clinical case series with different clinical and MRI presentation and discuss diagnostic and treatment challenges. During our screening period at neurological department we revealed 6 NS cases. Three of them have an atypical presentation. The first patient was misdiagnosed as acute disseminated encephalomyelitis, the second patient had hippocampal sclerosis and epileptic seizures. Another patient had cognitive decline and autoimmune encephalitis-like MRI lesions. We put an emphasis on widening of indication for lumbar puncture and NS tests in patients with syphilitic anamnesis and neurological manifestations.
Keywords: Atypical cases; CSF tests; MRI; Neurosyphilis.
Copyright © 2019 Elsevier Ltd. All rights reserved.
- Case Reports
- Anti-Bacterial Agents / therapeutic use
- Middle Aged
- Neurosyphilis / complications
- Neurosyphilis / diagnosis*
- Neurosyphilis / drug therapy
- Treponema pallidum
- Anti-Bacterial Agents
- Reference Manager
- Simple TEXT file
People also looked at
Review article, chatgpt in orthopedics: a narrative review exploring the potential of artificial intelligence in orthopedic practice.
- 1 IRCCS Istituto Ortopedico Galeazzi, Milan, Italy
- 2 Residency Program in Orthopedics and Traumatology, University of Milan, Milan, Italy
- 3 Department of Plastic Surgery, University of Pittsburgh Medical Center, Pittsburgh, PA, United States
- 4 Department of Orthopaedic, Trauma, and Reconstructive Surgery, RWTH University Medical Centre, Aachen, Germany
- 5 Department of Orthopedics and Trauma Surgery, Academic Hospital of Bolzano (SABES-ASDAA), Teaching Hospital of the Paracelsus Medical University, Bolzano, Italy
- 6 Dipartimento di Scienze Biomediche per la Salute, Università degli Studi di Milano, Milan, Italy
The field of orthopedics faces complex challenges requiring quick and intricate decisions, with patient education and compliance playing crucial roles in treatment outcomes. Technological advancements in artificial intelligence (AI) can potentially enhance orthopedic care. ChatGPT, a natural language processing technology developed by OpenAI, has shown promise in various sectors, including healthcare. ChatGPT can facilitate patient information exchange in orthopedics, provide clinical decision support, and improve patient communication and education. It can assist in differential diagnosis, suggest appropriate imaging modalities, and optimize treatment plans based on evidence-based guidelines. However, ChatGPT has limitations, such as insufficient expertise in specialized domains and a lack of contextual understanding. The application of ChatGPT in orthopedics is still evolving, with studies exploring its potential in clinical decision-making, patient education, workflow optimization, and scientific literature. The results indicate both the benefits and limitations of ChatGPT, emphasizing the need for caution, ethical considerations, and human oversight. Addressing training data quality, biases, data privacy, and accountability challenges is crucial for responsible implementation. While ChatGPT has the potential to transform orthopedic healthcare, further research and development are necessary to ensure its reliability, accuracy, and ethical use in patient care.
Musculoskeletal disorders affect millions of individuals worldwide each year and orthopedic surgeons often face challenging situations requiring quick and complex decisions. Furthermore, patients' education and compliance in orthopedics are essential in improving treatment outcomes and active participation in recovery ( 1 ).
Over the years, technological advancements have significantly influenced the practice of orthopedics, with the integration of artificial intelligence (AI) systems showing great potential in improving patient care and outcomes. In fact, this new imposing reality is developing exponentially in the healthcare sector, especially due to the improvement in computing power, the increase in health data, and the ability to access large sets of exploitable data ( 2 ). There are numerous stages of patient management where AI could play a useful role, ranging from the diagnostic to the therapeutic aspect. Among the various AI-based systems, ChatGPT, a language natural processing (LNP) technology developed by OpenAI (San Francisco, CA), was launched in November 2022.
ChatGPT is one of the LNP models based on the transformer architecture and trained on a vast corpus of textual data, enabling it to generate human-like responses to user questions in an interactive way. Its ability to understand and generate contextually relevant and coherent responses has led to its exploration and application in various sectors, including healthcare. In the field of orthopedics, this AI-based tool can provide clinical contributions to the complex decision-making process by facilitating information exchange with patients and providing accessible and accurate information to both healthcare professionals and patients themselves. The trends AI research following the launch of ChatGPT have recently been analyzed with the aim of identifying key developments and future directions. Alessandri-Bonetti et al. conducted a bibliometric analysis of the literature in the first 7 months since the introduction of ChatGPT until July 1st, 2023, collecting 724 articles ( 3 ). A significant increase in publications exploring ChatGPT use across various medical disciplines has been observed, especially in the medical field, suggesting a growing relevance of ChatGPT in the healthcare sector. Interestingly, a decrease in studies focused on ethical considerations has been noted, indicating a shift in research focus. The results highlight the increasing integration of ChatGPT in various medical disciplines, underscoring its expanding role in healthcare.
Among all areas of medicine, orthopedics deserves particular attention. Orthopedic conditions encompass a wide range of pathologies, including fractures, joint disorders, spinal deformities, and sports injuries. ChatGPT has the potential to serve as a clinical decision-support tool by providing clinicians with relevant information based on patient symptoms, medical history, and radiological findings. Its features can be helpful in differential diagnosis and suggest diagnostic tests or appropriate imaging modalities for further evaluation. Therapeutic recommendations in orthopedics are often based on evidence-based guidelines and clinical experience. AI technologies can optimize this process by assisting clinicians in synthesizing a vast amount of medical literature and providing updated therapeutic recommendations based on the specific characteristics of the patient and their condition. This can contribute to optimizing treatment plans, promoting adherence to evidence-based practices, and reducing variability in clinical decisions. Furthermore, ChatGPT could play a fundamental role in patient communication and education. Orthopedic conditions can often be complex, and patients often have numerous questions and concerns about their diagnosis, treatment options, and expected outcomes. ChatGPT can provide patients with reliable and understandable information, addressing their questions and alleviating their anxieties. Also, patients might enhance their knowledge and preparedness prior to surgeon's consult, potentially resulting not only in patient's readiness but also time saving for the physician. This can lead to improved patient satisfaction, interaction, and adherence to treatment plans. While this perspective could open up new opportunities for patients, it is likely dangerous to envision the use of ChatGPT as a substitute for the physical examination by a medical professional or specialist consultation.
Finally, ChatGPT can be a valuable tool for literature review and research in orthopedics, which is continuously evolving, with new studies and publications being released regularly. Keeping up with the latest evidence can be a challenge for clinicians and researchers. ChatGPT can assist in conducting literature searches, summarizing research articles, and identifying key findings, thus facilitating evidence-based practice and promoting knowledge translation. The significant impact of Artificial Intelligence in writing or assisting researchers has led several international scientific journals to require the declaration of whether AI software was used in writing an article. Indeed, despite the numerous potential advantages, it is essential to ensure scientific integrity and ethics in AI-assisted research and writing. Simultaneously, transparency regarding the use of AI in documents is a mandatory step towards genuine scientific responsibility.
Protecting this and many other aspects must be mandatory in approaching this pivotal shift in the medical and orthopedic world. The aim of this review is to provide a comprehensive overview of the use of ChatGPT in orthopedics, highlighting the pros and cons of each application. By synthesizing the available evidence, we hope to shed light on the strengths, limitations, and future implications of ChatGPT in enhancing patient care, clinical decision-making, and workflow optimization. The findings of this review will inform healthcare professionals, researchers, and policymakers about the current state of knowledge in this field and provide guidance for future research and implementation of ChatGPT in orthopedic practice.
2. Materials and methods
Studies were searched on PubMed database using the keywords “ChatGPT” OR “language natural processing” AND “Orthopaedics”. Last search was conducted on July 1st, 2023. Only studies describing the application of ChatGPT in orthopedics were included in the review. Studies involving the use of ChatGPT in orthopedic settings, such as clinical practice, patient education, decision support, and remote monitoring, will be considered. Exclusion criteria will include studies not relevant to orthopedics, non-English articles, and studies with inadequate information on the use of ChatGPT.
Two independent reviewers (R.G. and A.L.) performed the study selection, data extraction, and quality assessment. Any discrepancies will be resolved through consensus or consultation with a third reviewer (M.A.B.). The extracted data will include study characteristics, study design, and key findings.
A diverse range of studies on the use of ChatGPT in orthopedics was observed. The results are presented in a narrative synthesis, organized according to the different domains of orthopedic practice in which ChatGPT has been utilized. The main topics in which ChatGPT was tested were clinical decision-making, patient education, and workflow optimization. Table 1 provides key study characteristics.
Table 1 . Key study caractreristics.
4.1. exploring the diverse applications of chatgpt in orthopedics: from diagnosis to treatment planning.
The use of ChatGPT in the field of orthopedics has started to be explored in the scientific literature, with numerous articles discussing its potential. As reported in the work of Poduval et al. ( 4 ), it is now essential to understand and embrace robotics and AI, along with traditional clinical skills, in modern orthopedic practice. Since AI has the potential to be a positive and disruptive force in orthopedic surgery, orthopedic surgeons must accept and explore its possibilities. Indeed, advantageous prospects can be found in improving diagnostic accuracy, optimizing surgical planning, providing effective intraoperative assistance, and personalizing treatments. At the same time, the potential disruptive force of this technology must be monitored in areas such as data security, the need for continuous medical supervision, and the maintenance of medical ethics and integrity.
According to Cheng et al. ( 5 ), the main roles of ChatGPT can be found in scientific research, disease diagnosis, treatment options, preoperative planning, intraoperative support, and postoperative rehabilitation.The incredible potential of AI in orthopedic surgery is further discussed in the paper by Hernigou ( 6 ). According to the authors, the unique characteristic of AI that is well-suited to this medical field is its ability to analyze large amounts of data and generate useful information. With this feature, AI can not only assist in diagnosis, preoperative planning, or intraoperative guidance but also provide clinical decision support based on predictive analysis and personalized treatment plans.
Karnuta et al. ( 7 ) even compare the transformative potential of AI technology to historical advancements such as the introduction of metallic instruments and the Industrial Revolution. In this article as well, the authors hypothesize a real revolution in orthopedic practice in areas such as personalized patient care, image analysis, and surgical decision-making. To overcome the current limitations of ChatGPT in synthesizing complex orthopedic knowledge and answering intricate questions, the authors suggest that specialized training and exposure to orthopedic texts and manuscripts could enable AI systems to achieve higher performance levels and even surpass orthopedic exams. Although clinical applications are still lacking, and the AI technology still appears weak in real-life complex scenarios, according to the available reports, it is foreseeable that ChatGPT potential or future AI models will dramatically change orthopedics practice.
4.2. Empowering patients with AI: assessing the role of ChatGPT in providing reliable health information
In today's world, where patients have access to a vast amount of data (often not accurate and up-to-date), a crucial role could be played by this AI-based tool in patient information. In the paper by Dubin et al. ( 8 ), a comparison is made between the appropriateness and reliability of ChatGPT and Google web search as resources for patients seeking health information online. The study compares frequently asked questions (FAQs) related to total knee arthroplasty (TKA) and total hip arthroplasty (THA) obtained from both sources. Only 25% of the questions were similar when performing a Google web search and a search of ChatGPT for all search terms, with 13/20 Google results from commercial sites and 15/20 ChatGPT results from government sites. 11/20 numerical questions had different responses. ChatGPT provided heterogeneous questions and responses. In conclusion, it is not yet a reliable source of information for patients. More research is needed to determine its accuracy and reliability. Until then, patients should consult with a healthcare professional for medical questions or concerns.
4.3. Challenging ChatGPT
ChatGPT has been put to the test in various fields of medicine, and some have even attempted to “challenge” the AI-based ChatGPT model in the field of orthopedics, comparing it to human knowledge. The interesting work by Cuthbert ( 9 ) aimed to evaluate whether ChatGPT could pass Section 1 of the Fellowship of the Royal College of Surgeons (FRCS) examination in Traumatology and Orthopedic Surgery. The results demonstrated that ChatGPT achieved only 35.8%, significantly lower than the passing rate of FRCS and the average score obtained by human candidates at all levels of training. The main shortcomings of ChatGPT were identified in its inability to exercise higher-order judgment and the multilogical thinking required to pass the examination. These limitations should be recognized and publicized to ensure that clinicians are aware of them. The results of this study also underline the importance of critically assessing the reliability and limitations of artificial intelligence systems in the context of real-life complex scenarios. While ChatGPT has shown promise in generating contextually relevant text, its performance in a highly specialized and technical domain like orthopedic surgery has been insufficient. This suggests that AI models like ChatGPT may not necessarily possess the necessary expertise and clinical reasoning skills required for complex medical decisions. Additionally, the study revealed that ChatGPT failed to recognize its own limitations, providing incorrect explanations for questions it answered incorrectly. This represents a significant and dangerous limitation of this tool. Clinicians and educators should be cautious about relying solely on artificial intelligence systems for assessments or decisions without understanding their limitations and ensuring adequate human oversight. Adapting the training data and refining the model with specialized orthopedic knowledge could enhance its performance in this domain. Furthermore, efforts should be made to address the lack of contextual understanding exhibited by ChatGPT, as this is a crucial aspect of clinical decision-making. A recent study aimed to evaluate ChatGPT's performance in the Italian Residency Admission National Exam to assess its level of medical knowledge compared to graduate medical doctors in Italy ( 10 ). In June 2023, ChatGPT3 was employed to undertake this exam, which consists of a computer-based multiple-choice test comprising 140 questions, taken annually by all Italian medical graduates. The exam evaluates basic medical science knowledge and its application. ChatGPT's performance was compared with that of 15,869 medical graduates, revealing that ChatGPT answered 122 out of 140 questions correctly. The score ranked in the 98.8th percentile among the 15,869 medical graduates. Among the 18 incorrect answers, 10 related to direct questions about basic medical science knowledge, while 8 concerned applied clinical knowledge and reasoning through case presentations. Errors were logical (2 incorrect answers, ChatGPT motivated correctly the answer, but provided the wrong multiple-choice answer) and informational in nature (16 incorrect answers, ChatGPT provided incorrect answer and reasoning). Interestingly, all explanations for correct answers were deemed “appropriate.” Comparing with national statistics regarding the minimum score required to access each specialty, ChatGPT's performance demonstrated it would have qualified the candidate for any specialty. Thus, ChatGPT displayed competence in basic medical science knowledge and applied clinical knowledge. Further research should evaluate ChatGPT's impact and reliability in clinical practice.
4.4. ChatGPT in scientific literature: opportunities, challenges, and the imperative of ethical standards
There is another important aspect where ChatGPT is gaining traction, namely the field of scientific literature. The potential uses can vary widely, ranging from grammar correction and proofreading to planning the highlights of scientific articles. According to Bi et al. ( 11 ), the ability of ChatGPT to generate manuscript drafts and its potential to streamline the writing process should be acknowledged. However, concerns are raised about the accuracy of the generated content and the need for quality control and fact-checking. Ollivier et al. ( 12 ) discuss the problem of plagiarism and false content in scientific literature. We agree with the points raised by the authors, emphasizing the importance of maintaining high ethical standards and accuracy in scientific research. While large language models like ChatGPT have the potential to assist in text generation and information synthesis, it is essential to critically evaluate their results for scientific validity. The authors propose measures such as data sharing, improved training and education, and the development of technologies and tools to detect plagiarism and misconduct. The need to verify and corroborate the information generated by AI models, as well as the importance of ethical standards, transparency, and reliability in scientific research and publication, remain pressing. The role of human evaluation and critical thinking is still indispensable for the effective and responsible use of AI-generated content.
4.5. Cautions and recommendations
Given the aforementioned points, we feel obligated to provide cautions and recommendations for the interpretation of data derived from ChatGPT, as already shared by many authors of the aforementioned studies. New horizons and challenges such as data privacy, security, validation, and ethical considerations arise when ensuring responsible implementation of AI in orthopedic surgery. The responsible use of this tool must be based on an awareness of its limitations and biases. Foremost among them is the dangerous concept of AI hallucination ( 6 ). This phenomenon involves the possibility of generating incorrect responses but still providing confident and plausible-sounding explanations. The authors cite the following example: when asked to generate a report on an event after its last update, the chatbot falsely discusses the announcement but later admits that it has no information about this communication because it lacks temporal data availability. AI can rely on machine learning algorithms trained on extensive datasets to assess source credibility through reputation analysis and source consistency, aiming to identify potential patterns of misinformation or the spread of false information. Another tool at its disposal is spoken natural language analysis, which, through semantic and syntactic examination, can help recognize inaccurate or misleading information. Despite having these resources, AI hallucinations can be extremely perilous when critical analysis of obtained information is not conducted. Therefore, careful scrutiny is needed to avoid the inadvertent distribution of misleading or inaccurate medical knowledge.
Another aspect that deserves caution is the potential risk of bias in ChatGPT's responses. The generated answers could be influenced by the training data, which may reflect biases or trends in the original texts that are not necessarily accurate or up-to-date. This could manifest as formulating biased or unrepresentative recommendations or diagnoses in orthopedics. Therefore, we emphasize the importance of conducting a critical assessment of the responses and considering possible measures to mitigate any bias. Another challenge may lie in the demand for Structured Content Generation by ChatGPT. In the field of orthopedics, this could translate into the creation of orthopedic medical reports, which require strict formatting and organization of information. In this case, we also recommend a careful manual review of the generated documents to ensure the proper structuring of data.
Other peculiar elements that deserve attention are described in the article by Karnuta et al. ( 7 ), such as the “garbage in, garbage out” principle, emphasizing the importance of ensuring high-quality and unbiased data as input for AI systems to avoid perpetuating biases and misinformation. The same article also discusses the responsibility and obligation to ensure robust safety mechanisms and clear roles for stakeholders in the event of system malfunctions and harm to patients ( 7 ). These models may not possess in-depth domain-specific knowledge and may lack the ability to apply higher-order judgment and reasoning, especially in complex medical contexts ( 13 ). Aspects such as transparency, responsibility, and thorough evaluation of AI systems need to be sought and improved to ensure the reliability and quality of the generated results ( 13 ). Lastly, there are other elements worth mentioning, such as data privacy, quality control, biases in training data, and the challenge of authors' attribution ( 5 , 14 ). Therefore, careful regulation and ethical use of tools like ChatGPT in orthopedics and medicine seem necessary.
Finally, we recommend caution in managing multiple tasks simultaneously. Although ChatGPT can handle a wide range of tasks, this could pose limitations. In the field of orthopedics, a physician may have to address multiple questions simultaneously in a single interaction with ChatGPT. This may necessitate greater care in formulating questions and interpreting responses to ensure no confusion and thus provide accurate answers.
With these considerations, physicians should actively shape the trajectory of AI, providing feedback to regulatory bodies and developers, promoting dialogue, and ensuring a thorough examination of the implications of AI implementation in clinical practice ( 7 ).
4.6. Strengths and limitations
To the best of our knowledge, this is the first review of ChatGPT in the orthopedic field. This paper provides a comprehensive overview of the use of ChatGPT in orthopedics, covering various aspects such as clinical decision-making, patient education, workflow optimization, and scientific literature. The present study presents both the potential benefits and limitations of using ChatGPT, highlighting the need for caution, ethical considerations, and human oversight.
This study also presents several limitations. First of all, it is a narrative review. So, although the review mentions the use of independent reviewers and quality assessment tools, it does not follow a standard systematic review methodology, such as a predefined protocol or PRISMA guidelines. Secondly, there is a lack of critical appraisal. This narrative review does not provide an evaluation of the quality or risk of bias of individual studies. A critical appraisal of the included studies would allow readers to assess the strength of the evidence presented. In conclusion, while this narrative review provides a comprehensive overview of the potential applications of ChatGPT in orthopedics and highlights the need for caution and ethical considerations, its limitations as a non-systematic review and lack of critical appraisal of included studies should be considered when interpreting the findings.
The integration of AI technologies, including ChatGPT, holds tremendous promise for transforming orthopedic healthcare. Although the potential applications of ChatGPT in orthopedics are promising, several challenges and considerations need to be addressed. The reliability and accuracy of the responses generated by ChatGPT depend on the quality of the training data and algorithms used. It is essential to ensure that the language model is trained on diverse and high-quality orthopedic data to minimize the risk of bias and incorrect recommendations. Furthermore, ethical and legal aspects of AI use in healthcare, such as data privacy, security, and accountability, must be carefully addressed to ensure patient confidentiality and trust.
Addressing the challenges and considerations associated with its use is crucial to ensure the reliability, accuracy, and ethical implementation of this technology. Ongoing research and development in this field will pave the way for the integration of ChatGPT and other artificial intelligence systems in orthopedics, benefiting both patients and healthcare providers.
RG: Conceptualization, Methodology, Visualization, Writing – original draft, Writing – review & editing. MA: Conceptualization, Investigation, Writing – original draft. AL: Conceptualization, Investigation, Writing – review & editing. FM: Conceptualization, Writing – review & editing. NR: Investigation, Methodology, Writing – review & editing. LM: Methodology, Supervision, Writing – review & editing. GP: Supervision, Writing – review & editing.
The author(s) declare financial support was received for the research, authorship, and/or publication of this article.
This research was funded by the Italian Ministry of Health—Ricerca Corrente. The Italian Ministry of Health has also paid the APCs (Article Processing Charges).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
1. Loftus TJ, Tighe PJ, Filiberto AC, Efron PA, Brakenridge SC, Mohr AM, et al. Artificial intelligence and surgical decision-making. JAMA Surg . (2020) 155:148–58. doi: 10.1001/jamasurg.2019.4917
PubMed Abstract | CrossRef Full Text | Google Scholar
2. Kaul V, Enslin S, Gross SA. History of artificial intelligence in medicine. Gastrointest Endosc . (2020) 92:807–12. doi: 10.1016/j.gie.2020.06.040
3. Alessandri-Bonetti M, Liu HY, Giorgino R, Nguyen VT, Egro FM. The first months of life of ChatGPT and its impact in healthcare: a bibliometric analysis of the current literature. Ann Biomed Eng . (2023. doi: 10.1007/s10439-023-03325-8
CrossRef Full Text | Google Scholar
4. Poduval M, Ghose A, Manchanda S, Bagaria V, Sinha A. Artificial intelligence and machine learning: a new disruptive force in orthopaedics. Indian J Orthop . (2020) 54:109–22. doi: 10.1007/s43465-019-00023-3
5. Cheng K, Li Z, Li C, Xie R, Guo Q, He Y, et al. The potential of GPT-4 as an AI-powered virtual assistant for surgeons specialized in joint arthroplasty. Ann Biomed Eng . (2023) 51:1366–70. doi: 10.1007/s10439-023-03207-z
6. Hernigou P, Scarlat MM. Two minutes of orthopaedics with ChatGPT: it is just the beginning; it’s going to be hot, hot, hot!. Int Orthop . (2023) 47:1887–93. doi: 10.1007/s00264-023-05887-7
7. Karnuta JM. CORR Insights®: can artificial intelligence pass the American board of orthopaedic surgery examination? Orthopaedic residents versus ChatGPT. Clin Orthop Relat Res . (2023) 481:1631–3. doi: 10.1097/CORR.0000000000002741
8. Dubin JA, Bains SS, Chen Z, Hameed D, Nace J, Mont MA, et al. Using a google web search analysis to assess the utility of ChatGPT in total joint arthroplasty. J Arthroplasty . (2023) 38:1195–202. doi: 10.1016/j.arth.2023.04.007
9. Cuthbert R, Simpson AI. Artificial intelligence in orthopaedics: can chat generative pre-trained transformer (ChatGPT) pass section 1 of the fellowship of the royal college of surgeons (trauma & orthopaedics) examination? Postgrad Med J . (2023):qgad053. doi: 10.1093/postmj/qgad053
10. Alessandri Bonetti M, Giorgino R, Gallo Afflitto G, De Lorenzi F, Egro FM. How does ChatGPT perform on the Italian residency admission national exam compared to 15,869 medical graduates? Ann Biomed Eng . (2023). doi: 10.1007/s10439-023-03318-7
11. Bi AS. What’s important: the next academic-ChatGPT AI? J Bone Joint Surg Am . (2023). doi: 10.2106/JBJS.23.00269
12. Ollivier M, Pareek A, Dahmen J, Kayaalp ME, Winkler PW, Hirschmann MT, et al. A deeper dive into ChatGPT: history, use and future perspectives for orthopaedic research. Knee Surg Sports Traumatol Arthrosc . (2023) 31:1190–2. doi: 10.1007/s00167-023-07372-5
13. Kunze KN, Jang SJ, Fullerton MA, Vigdorchik JM, Haddad FS. What’s all the chatter about? Bone Joint J . (2023) 105-B:587–9. doi: 10.1302/0301-620X.105B6.BJJ-2023-0156
14. Parsa A, Ebrahimzadeh MH. ChatGPT in medicine; a disruptive innovation or just one step forward? Arch Bone Jt Surg . (2023) 11:225–6. doi: 10.22038/abjs.2023.22042
Keywords: orthopedics, artificial intelligence, AI, ChatGPT, patient education, clinical decision-making
Citation: Giorgino R, Alessandri-Bonetti M, Luca A, Migliorini F, Rossi N, Peretti GM and Mangiavini L (2023) ChatGPT in orthopedics: a narrative review exploring the potential of artificial intelligence in orthopedic practice. Front. Surg. 10:1284015. doi: 10.3389/fsurg.2023.1284015
Received: 27 August 2023; Accepted: 16 October 2023; Published: 1 November 2023.
© 2023 Giorgino, Alessandri-Bonetti, Luca, Migliorini, Rossi, Peretti and Mangiavini. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Riccardo Giorgino [email protected]
This article is part of the Research Topic
Artificial Intelligence-Based Multimodal Prediction Modeling in Orthopedic Surgery
Medical Faculty of University of Peradeniya Collaborates with Moscow City Clinical Hospital
A video-conference was held between the Medical Faculty of University of Peradeniya and Moscow City Clinical 52 to discuss about specified areas and aspects for further collaboration in experts’ opinion exchange, online interaction, research and education, and signing of an MOU between the University and the Clinical Hospital.
In his introductory remarks, Ambassador Prof. M.D. Lamawansa underlined the first steps to establish links between the two institutions. Dean of the Medical Faculty Prof. Asiri Abeyagunawardena spoke about the main activity of the Center for Education, Research and Training on Kidney Diseases of the University with specific attention to the prevalence of chronic kidney disease of unknown origin in the country.
Prof. Thushara Kudagammana presented on paediatric allergic diseases and asthma in Sri Lanka, highlighting the pressing issues and developments with regard to diagnostics and therapeutic options, training and research while Dr. Champa Ratnatunga made a presentation on covering the general information on Laboratory Immunology, laboratory infrastructure, and specified the areas where diagnostics and research capabilities need to be further developed.
From Russian side, Dr. Darya Fomina briefed about the clinical and scientific resources of the regional center for allergy and immunology, focal problems and achievements, and ongoing projects on international cooperation and Dr. Oleg Kotenko spoke about the history of Moscow City Clinical and Scientific Center of Nephrology, Kidney Transplant Pathology and Urology, developed methodology, cumulative experience, structure and innovative multidisciplinary telemedicine approaches of the center.
During the meeting the participants expressed their readiness to cooperate in the possible areas of mutual benefit.
Moscow City Clinical Hospital was represented by its President Prof. Vladimir Vtorenko and his experts. From the University of Peradeniya, Dean of the Medical Faculty Prof. Asiri Abeygunawardana and his professional team took part in the discussion.
Ambassador Prof. M.D. Lamawansa and the Embassy team facilitated the meeting.
පේරාදෙණිය විශ්වවිද්යාලයේ වෛද්ය පීඨය හා මොස්කව් නුවර සායනක රෝහල අතර
වෛද්ය පර්යේෂණ හා ප්රතිකාර පිළිබඳ අත්දැකිම් බෙදා ගැනීමේ වැඩසටහනක්
ශ්රී ලංකාව හා මොස්කව් නුවර වෛද්ය පර්යේෂණ හා ප්රතිකාර පිළිබඳ අත්දැකිම් බෙදා ගැනීමේ වැඩ සටහනක් ඉකුත් සතියේ ආරමිභ විය. මෙහි ප්රථම වැඩසටහන ලෙස පේරාදෙණිය විශ්වවිද්යාලයේ වෛද්ය පීඨය හා මොස්කව් නුවර අංක 52 සායනක රෝහල අතර වීඩියෝ තාක්ෂණය ඔස්සේ ළමා රෝග, වකුගඩු බද්ධ කීරිම, ප්රතිශක්තිකරණය හා රසායනාගාර කටයුුතු ආදි විෂයයන් රැසක අත්දැකීම් බෙදා ගනු ලැබීය.
රුසියානු සමුහාණ්ඩුවේ ශ්රී ලංකා තානාපති විශේෂඥ වෛද්ය මහචාර්ය එම්.ඩි. ළමාවංශ මහතා ගේ මෙහෙයවීම අනුව මෙම වෛද්ය පර්යේෂණ හා ප්රතිකාර අත්දැකීම් හුවමාරු වැඩසටහන ආරම්භ කර ඇත. ඉකුත් 5දා පැවැති මෙහි මුල්ම වැඩ සටහනට පේරාදෙණිය විශ්වවිද්යාලයේ වෛද්ය පීඨයේ පීඨාධිපති මහචාර්ය ආසිරි අබේගුණවර්ධන ළමා අසාත්මිකතා හා ප්රතිශක්ති විද්යාව පිළිබඳ විශේෂඥ වෛද්ය තුෂාර කුඩගමමන, ළමා වෘක්කවේද වෛද්ය ෂෙනාල් තල්ගහගොඩ, වැඩිහිටි වෘක්කවේද වෛද්ය රංජිත් අබේසේකර, වකුගඩු ව්යාධිවේදි වෛද්ය සුලෝචනා විජේතුංග, ප්රතිශක්ති විද්යාව පිළිබඳ විශේෂඥ වෛද්ය මහාචාර්ය සංජය අධිකාරි, ක්ෂුද්රජීව විද්යාව හා රසායනාගාර ප්රිශක්ති විද්යාව පිළිබඳ විශේෂඥ වෛද්ය චම්පා රත්නතුංග යන මහත්ම මහත්මීන් සමඟ මොස්කව් අංක 52 සායනික රෝහලේ සභාපති මහාචාර්ය විලැඩිමීර් විටෝරෙන්කෝ, අසාත්මික හා ප්රතිශක්තිකරණ මධ්යස්ථානයේ ප්රධානි වෛද්ය චාරියා ෆෝමිනා, වකුගඩු බද්ධ කීරීමේ ව්යාධි විද්යාව පිළිබඳ විද්යාත්මක මධ්යස්ථානයේ ප්රධානි ඔලෙක් කොටෙන්කෝ ජාත්යන්තර සහයෝගිතා දෙපාර්තමේන්තුවේ ප්රධානි වෛද්ය වට්යානා කිරිලෝවා යන මහත්ම මහත්මින් මෙම වැඩ සටහනට සම්බන්ධ වී සිටියහ.
An official website of the United States government
The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.
The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.
- Account settings
- Advanced Search
- Journal List
- J Grad Med Educ
- v.8(5); 2016 Dec
Global Health and Graduate Medical Education: A Systematic Review of the Literature
Associated data, background .
Global health (GH) interest is increasing in graduate medical education (GME). The popularity of the GH topic has created growth in the GME literature.
The authors aim to provide a systematic review of published approaches to GH in GME.
We searched PubMed using variable keywords to identify articles with abstracts published between January 1975 and January 2015 focusing on GME approaches to GH. Articles meeting inclusion criteria were evaluated for content by authors to ensure relevance. Methodological quality was assessed using the Medical Education Research Study Quality Instrument (MERSQI), which has demonstrated reliability and validity evidence.
Overall, 69 articles met initial inclusion criteria. Articles represented research and curricula from a number of specialties and a range of institutions. Many studies reported data from a single institution, lacked randomization and/or evidence of clinical benefit, and had poor reliability and validity evidence. The mean MERSQI score among 42 quantitative articles was 8.87 (2.79).
There is significant heterogeneity in GH curricula in GME, with no single strategy for teaching GH to graduate medical learners. The quality of literature is marginal, and the body of work overall does not facilitate assessment of educational or clinical benefit of GH experiences. Improved methods of curriculum evaluation and enhanced publication guidelines would have a positive impact on the quality of research in this area.
Interest in global health (GH) education is a relatively recent phenomenon in academic medicine. Over the last 2 decades interest in GH in graduate medical education (GME) has steadily grown, with many residencies currently offering curricula and experiences in GH. 1 , 2 The popularity of GH, especially in disciplines such as emergency medicine, family medicine, pediatrics, and internal medicine, has led to similar growth in the literature on GH education in GME settings. 3 While there is no single, agreed-on definition of global health, our understanding rests on a definition provided by Koplan et al, 4 that GH is “an area for study, research, and practice that places a priority on improving health and achieving equity in health for all people worldwide.”
While there is mounting agreement about the merits of GH, there is little consensus about how to educate the next generation of residents and fellows interested in GH. 5 Preparing an effective workforce to meet worldwide clinical and educational priorities requires research into GH-related educational needs in GME. 6
Academic medical centers have aimed to improve educational experiences by standardizing GH education through establishing curricula and formal competency requirements, including the development of GH electives, tracks, and didactic study. 7 , 8 Fellowship training in GH is also available, especially in pediatrics, internal medicine, and emergency medicine. 9
Additionally, authors have proposed distinct educational strategies 10 and standardized medical school–based curriculum guides. 11 At present, however, course offerings in GH tend to vary by academic centers. There are few previously published reviews synthesizing the literature on the overall quality and effectiveness of specific educational approaches in GH education in GME.
Given the lack of existing summaries, we aim to provide a systematic review of published approaches to GH in the GME literature. Our aim is to provide insight on the current state of the literature on GH education, synthesize current trends, and to formulate recommendations on the future development of GH education through analysis of the variety of developed curricula, teaching approaches, and evaluation methods.
We planned, executed, and reported this systematic review in adherence to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). 12
We conducted a systematic review of the literature via PubMed using variable terms to identify articles with abstracts published between January 1975 and January 2015 focused on GH and GME. Key search terms included variable iterations of the following keywords: medical, education, graduate, international, global, health, training, curriculum, residency, course, standards, electives, and competencies.
Studies were limited to those focused on GH education in GME, specifically focused on clinicians in training including residents and fellows, and excluded studies focused on medical student learners and learners in other allied health fields (such as pharmacy, nursing, or public health).
Studies were additionally required to meet the following inclusion criteria: English language; inclusion of an abstract; representative of original research or programming (nonabstract opinion pieces or letters to the editor were excluded); program description had to be part of a GME (residency or fellowship) program centered in the United States or Canada; and majority of study participants had to be enrolled in a GME program (residency or fellowship). We considered all articles regardless of methodological approach, including both qualitative and/or quantitative methodologies.
Title and Abstract Review
Both study authors reviewed the titles and abstracts of articles. If the abstract was insufficient to provide decision for inclusion, then the full text was reviewed independently by each of the 2 authors. In the event that the 2 authors disagreed on inclusion, the full text was re-reviewed, the article discussed, and consensus agreement was reached based on the predetermined inclusion or exclusion criteria.
Study Review/Data Extraction and Analysis
We developed a data extraction form that would allow for easy analysis based on the Best Evidence Medical Education Collaboration. 13
The form was initially piloted in a review of 16 randomly selected studies (each reviewed by both authors) to confer ease of relevance and consistency in filling out the form. Data abstracted from the studies included in this review consisted of 4 domains: (1) journal (name and impact factor); (2) study characteristics (publication year, study design, number, name, and location of participating institution[s]); (3) participants (number, level of training, profession, and medical specialty); and (4) outcome measures.
Methodological quality among articles employing a quantitative analysis was assessed using the Medical Education Research Study Quality Instrument (MERSQI), which has demonstrated relevance and validity. 14 The MERSQI score was originally designed to evaluate methodological quality of medical education research. It consists of 10 items, reflecting 6 domains of study quality: study design, sampling, type of data, validity, data analysis, and outcomes. 14 MERSQI scores were calculated for all quantitative studies included according to the methods described by the authors of the instrument. Each article meeting inclusion criteria was evaluated for content and scored by both authors to ensure relevance with possible summed scores ranging from 4.5 to 18.
Educational outcomes were also classified using Kirkpatrick's hierarchy. 15 Kirkpatrick level (KL) 0 provides no assessment of impact; KL-1 assesses learner reaction; KL-2 assesses attitude and knowledge or skills; KL-3 assesses changes in behavior; and KL-4 assesses changes in patient or systems-based outcomes.
Interrater reliability was determined for the elements of the MERSQI scores by calculating an interrater correlation coefficient. Disagreements between raters in assigning quantitative scores were resolved by discussion (only after the interrater reliability was determined), re-review of the article, and achievement of a final consensus rating. A consensus was derived in all field types for each of the articles. We used the consensus mean and median MERSQI scores with standard deviations/interquartile range to describe the overall quality of included studies. All analyses were performed in Stata version 13 (StataCorp LP, College Station, TX).
Initial broad search criteria led to 7113 unique articles ( figure ). This number was then refined to 496 based on title and abstract analysis alone. Further full article review led to 69 articles meeting the inclusion criteria, and 42 of them meeting criteria for MERSQI analysis ( figure ; article list provided as online supplemental material).
Selection Process Used in Systematic Review of Global Health Education in Graduate Medical Education Articles (1975–2014)
Abbreviation: MERSQI, Medical Education Research Study Quality Instrument.
Original work was published in a large variety of journals, including general GME and clinical specialty publications. The average impact factor of the journals in which the articles were published was 2.44 (SD = 1.3). The majority of scholarship was recent, with 97% (n = 67) published after 2000, and more than 70% (n = 50) published since 2010 ( table 1 ). Institutional representation was variable, with the majority of articles arising from university-affiliated academic medical centers (n = 65, 94%). Research meeting inclusion criteria represented 13 different medical specialties and subspecialties. Among studies that reported the number of participants (n = 45), the median (interquartile range) number was 26 (13–66).
Baseline Characteristics of Articles in a Systematic Review of Global Health Education in Graduate Medical Education (January 1975–January 2015)
A total of 61 articles reported on curriculum development and evaluation with the majority focused on clinical electives (n = 36). Further educational approaches included health tracks (n = 9), fellowships (n = 6), didactic lectures (n = 4), and web- and simulation-based training (each n = 1).
Of the 69 articles, 27 articles were qualitative studies, and 42 used quantitative methods. Of the 42 studies that used quantitative methodology, the most common study designs were single group (n = 30, 71%), with comparison groups used in a small number of studies (n = 9, 21%; table 2 ). No randomized controlled trials were represented in the dataset.
MERSQI Scores for Each of 10 Domains in Systematic Review of Global Health Education in Graduate Medical Education Articles (January 1975–January 2015)
Many studies lacked multiple institution sampling (n = 31, 74%), and measures beyond self-assessment were used in a minority of publications (n = 13, 31%). Additionally, there was little reported validity evidence for instruments used in studies, with 64% of included studies (n = 27) having a MERSQI summed validity score of 1 or less.
Analysis for the majority of articles was appropriate, given the study type (n = 38, 91%), but the sophistication of analysis was often limited to descriptive reporting only. Evidence supporting clinical benefit was also lacking, as no studies reported on patient- or health-related outcomes. The majority of publications reported on the following subjective outcomes: satisfaction, attitudes, perception, opinions, and general facts (n = 29, 69%). A minority assessed knowledge/skills (n = 7, 17%) and behaviors (n = 6, 14%).
The mean consensus MERSQI score was 8.87 ± 2.79 (±SD) out of a possible score of 18. Although overall study quality was found to be poor, there were 4 articles with MERSQI scores above 13 that provided evidence of good study quality. 16 – 19 The 4 articles were from 4 separate institutions, representing 4 distinct medical specialties, with the participant number ranging from 19 to 298. The articles were similar in that all reported objective data measures and provided an analysis that was beyond just descriptive and was appropriate for the data type.
For the scoring on the MERSQI, interclass correlation coefficient was excellent, with perfect agreement (interclass correlation coefficient = 1.0 [SD = 0.00]) for 5 items in the 10-item MERSQI scale. Items with limited disagreement included institutions (SD = 0.005), response rate (SD = 0.012), validity content (SD = 0.012), appropriateness of analysis (SD = 0.006), and sophistication of analysis (SD = 0.012). There were no items with some disagreement.
When evaluating Kirkpatrick's hierarchy of outcomes in studies that reported on an educational intervention, 81.2% (56 of 69) reported Level 1 outcomes (“learner satisfaction”), and only 10.1% (7 of 69) and 8.7% (6 of 69) of studies reported Level 2 (“change in attitude, knowledge, or skills”) and Level 3 (“change in behaviors”) outcomes, respectively. No studies reported Level 4 outcomes (“change in results”).
Qualitative studies offered similar findings to the quantitative studies, with individuals perceiving learning as generally positive, and analysis limited to perceptions and attitudes of the benefits of GH curricula and experiences.
The sizable increase of GH education publications after 2010 correlates with growing interest from residents and medical students in the United States and elsewhere. 1 This rapid growth may have contributed to heterogeneity in curricula, with no single “accepted” strategy for teaching GH in GME. The quality of analysis of the existing educational platforms is less than desirable, with particular weakness in reporting on the validity evidence of evaluation instruments. These findings corroborate other medical education studies that have found substandard methodology and validity reporting. 14 , 20 , 21
We sought to add to the conceptualization of the quality of the GH literature by reporting the impact factor of the journals in which articles were published. For the articles in our review, the average impact factor of the journals in which they were published was higher than the median impact factor for education journals as whole (0.902). 22
Deficiencies in medical education research are widely acknowledged, and the GH literature does not differ. Global health–related articles have a lower mean MERSQI score than previously studied medical education manuscripts (8.9 versus 10.7). 23 Our study found that the majority of studies were single institution, reducing generalizability, and the lack of comparison group in many of these studies weakens the data.
When assessed via Kirkpatrick's hierarchy of outcomes, the majority of studies reported Level 1 outcomes (learner satisfaction only), 15 and none of the representative articles reported on patient- or health-related outcomes. Moreover, the conclusions that study authors offered regarding the success or weakness of educational opportunities rested solely on learners' perceptions of their experience, rather than other, more robust outcomes.
We found a large and diverse number of educational approaches to GME global health. Though models of didactic learning and experiential learning appear to predominate, similar to prior research, there appears to be no agreed on consensus regarding what constitutes a successful technique for teaching GH to residents and fellows. The highest-quality articles reinforce the relationship between learning, often using core competencies and practice. The current crop of GME GH learners hail from the millennial generation, and the current model of didactic learning may need to change. The educational approach moving forward should share a common framework, including core competencies that incorporate generational preferences for e-learning technologies, and techniques such as “flipping the classroom,” in which the typical homework and lecture elements are reversed. 24 , 25
Recent proposals have reinforced the responsibility of sponsoring organizations in GH training experiences, with a need to focus on education and local needs, and the priorities of host institutions. 26 We have argued a similar position elsewhere, and promote efforts to focus on local needs, as well as evaluate local outcomes in the analysis of approaches to GH educational research. 27 The linking of patient outcomes to trainee performance has been proposed previously. 28 Examples of local outcomes include data on changes in number and type of patients seen, effectiveness of resident participation in clinical care, and changes in mortality and morbidity related to local demographics.
Our analysis of research on the current status of GH and medical education has several limitations, beginning with only including English-language articles from North American institutions indexed in PubMed. Second, while the authors were initially blinded to each other's scoring of the articles, they were not blinded to author(s), title, or journal name of the articles. Third, the authors used the MERSQI to assess study quality, and this metric does not encompass all aspects of study quality or allow for comment on an article's conceptual framework or the importance of the research question. Fourth, we reported on aspects of the journals in which articles were published, including the impact factor. While a general metric of the relative importance of a journal, impact factor does not necessarily serve as a direct indication of a given article's importance in its field.
In exploring the GH medical education literature, we identified heterogeneity in published studies, and 3 areas lacking in rigor and methodological quality. First, few of the studies included a comparison group. Second, studies need to be explicit about report validity evidence of evaluation instruments. Third, there is a need for more robust methods of GH curriculum evaluation, with a focus on levels 3 and 4 of Kirkpatrick's hierarchy of educational evaluation outcomes.
- Next Article
The Literature Review Defined
Purpose and importance of the literature review, approaching the literature review, locating and organizing the literature, getting organized, knowing when to say when, putting it all together, the literature review: a foundation for high-quality medical education research.
- Article contents
- Figures & tables
- Supplementary Data
- Peer Review
- Open the PDF for in another window
- Guest Access
- Get Permissions
- Cite Icon Cite
- Search Site
Lauren A. Maggio , Justin L. Sewell , Anthony R. Artino; The Literature Review: A Foundation for High-Quality Medical Education Research. J Grad Med Educ 1 July 2016; 8 (3): 297–303. doi: https://doi.org/10.4300/JGME-D-16-00175.1
Download citation file:
- Ris (Zotero)
- Reference Manager
A literature review forms the basis for high-quality medical education research and helps maximize relevance, originality, generalizability, and impact.
A literature review provides context, informs methodology, maximizes innovation, avoids duplicative research, and ensures that professional standards are met.
Literature reviews take time, are iterative, and should continue throughout the research process.
Researchers should maximize the use of human resources (librarians, colleagues), search tools (databases/search engines), and existing literature (related articles).
Keeping organized is critical.
Web of Science a
Education Resource Information Center (ERIC)
Cumulative Index of Nursing & Allied Health (CINAHL) a
a These are subscription resources. Researchers should check with their librarian to determine their access rights.
Despite a surge in published scholarship in medical education 1 and rapid growth in journals that publish educational research, manuscript acceptance rates continue to fall. 2 Failure to conduct a thorough, accurate, and up-to-date literature review identifying an important problem and placing the study in context is consistently identified as one of the top reasons for rejection. 3 , 4 The purpose of this editorial is to provide a road map and practical recommendations for planning a literature review. By understanding the goals of a literature review and following a few basic processes, authors can enhance both the quality of their educational research and the likelihood of publication in the Journal of Graduate Medical Education ( JGME ) and in other journals.
In medical education, no organization has articulated a formal definition of a literature review for a research paper; thus, a literature review can take a number of forms. Depending on the type of article, target journal, and specific topic, these forms will vary in methodology, rigor, and depth. Several organizations have published guidelines for conducting an intensive literature search intended for formal systematic reviews, both broadly (eg, PRISMA) 5 and within medical education, 6 and there are excellent commentaries to guide authors of systematic reviews. 7 , 8
Such work is outside the scope of this article, which focuses on literature reviews to inform reports of original medical education research. We define such a literature review as a synthetic review and summary of what is known and unknown regarding the topic of a scholarly body of work, including the current work's place within the existing knowledge . While this type of literature review may not require the intensive search processes mandated by systematic reviews, it merits a thoughtful and rigorous approach.
An understanding of the current literature is critical for all phases of a research study. Lingard 9 recently invoked the “journal-as-conversation” metaphor as a way of understanding how one's research fits into the larger medical education conversation. As she described it: “Imagine yourself joining a conversation at a social event. After you hang about eavesdropping to get the drift of what's being said (the conversational equivalent of the literature review), you join the conversation with a contribution that signals your shared interest in the topic, your knowledge of what's already been said, and your intention.” 9
The literature review helps any researcher “join the conversation” by providing context, informing methodology, identifying innovation, minimizing duplicative research, and ensuring that professional standards are met. Understanding the current literature also promotes scholarship, as proposed by Boyer, 10 by contributing to 5 of the 6 standards by which scholarly work should be evaluated. 11 Specifically, the review helps the researcher (1) articulate clear goals, (2) show evidence of adequate preparation, (3) select appropriate methods, (4) communicate relevant results, and (5) engage in reflective critique.
Failure to conduct a high-quality literature review is associated with several problems identified in the medical education literature, including studies that are repetitive, not grounded in theory, methodologically weak, and fail to expand knowledge beyond a single setting. 12 Indeed, medical education scholars complain that many studies repeat work already published and contribute little new knowledge—a likely cause of which is failure to conduct a proper literature review. 3 , 4
Likewise, studies that lack theoretical grounding or a conceptual framework make study design and interpretation difficult. 13 When theory is used in medical education studies, it is often invoked at a superficial level. As Norman 14 noted, when theory is used appropriately, it helps articulate variables that might be linked together and why, and it allows the researcher to make hypotheses and define a study's context and scope. Ultimately, a proper literature review is a first critical step toward identifying relevant conceptual frameworks.
Another problem is that many medical education studies are methodologically weak. 12 Good research requires trained investigators who can articulate relevant research questions, operationally define variables of interest, and choose the best method for specific research questions. Conducting a proper literature review helps both novice and experienced researchers select rigorous research methodologies.
Finally, many studies in medical education are “one-offs,” that is, single studies undertaken because the opportunity presented itself locally. Such studies frequently are not oriented toward progressive knowledge building and generalization to other settings. A firm grasp of the literature can encourage a programmatic approach to research.
Considering these issues, journals have a responsibility to demand from authors a thoughtful synthesis of their study's position within the field, and it is the authors' responsibility to provide such a synthesis, based on a literature review. The aforementioned purposes of the literature review mandate that the review occurs throughout all phases of a study, from conception and design, to implementation and analysis, to manuscript preparation and submission.
Planning the literature review requires understanding of journal requirements, which vary greatly by journal ( table 1 ). Authors are advised to take note of common problems with reporting results of the literature review. Table 2 lists the most common problems that we have encountered as authors, reviewers, and editors.
Sample of Journals' Author Instructions for Literature Reviews Conducted as Part of Original Research Article a
Common Problem Areas for Reporting Literature Reviews in the Context of Scholarly Articles
Three resources may facilitate identifying relevant literature: human resources, search tools, and related literature. As the process requires time, it is important to begin searching for literature early in the process (ie, the study design phase). Identifying and understanding relevant studies will increase the likelihood of designing a relevant, adaptable, generalizable, and novel study that is based on educational or learning theory and can maximize impact.
A medical librarian can help translate research interests into an effective search strategy, familiarize researchers with available information resources, provide information on organizing information, and introduce strategies for keeping current with emerging research. Often, librarians are also aware of research across their institutions and may be able to connect researchers with similar interests. Reaching out to colleagues for suggestions may help researchers quickly locate resources that would not otherwise be on their radar.
During this process, researchers will likely identify other researchers writing on aspects of their topic. Researchers should consider searching for the publications of these relevant researchers (see table 3 for search strategies). Additionally, institutional websites may include curriculum vitae of such relevant faculty with access to their entire publication record, including difficult to locate publications, such as book chapters, dissertations, and technical reports.
Strategies for Finding Related Researcher Publications in Databases and Search Engines
Search Tools and Related Literature
Researchers will locate the majority of needed information using databases and search engines. Excellent resources are available to guide researchers in the mechanics of literature searches. 15 , 16
Because medical education research draws on a variety of disciplines, researchers should include search tools with coverage beyond medicine (eg, psychology, nursing, education, and anthropology) and that cover several publication types, such as reports, standards, conference abstracts, and book chapters (see the box for several information resources). Many search tools include options for viewing citations of selected articles. Examining cited references provides additional articles for review and a sense of the influence of the selected article on its field.
Once relevant articles are located, it is useful to mine those articles for additional citations. One strategy is to examine references of key articles, especially review articles, for relevant citations.
As the aforementioned resources will likely provide a tremendous amount of information, organization is crucial. Researchers should determine which details are most important to their study (eg, participants, setting, methods, and outcomes) and generate a strategy for keeping those details organized and accessible. Increasingly, researchers utilize digital tools, such as Evernote, to capture such information, which enables accessibility across digital workspaces and search capabilities. Use of citation managers can also be helpful as they store citations and, in some cases, can generate bibliographies ( table 4 ).
Researchers often ask how to know when they have located enough citations. Unfortunately, there is no magic or ideal number of citations to collect. One strategy for checking coverage of the literature is to inspect references of relevant articles. As researchers review references they will start noticing a repetition of the same articles with few new articles appearing. This can indicate that the researcher has covered the literature base on a particular topic.
In preparing to write a research paper, it is important to consider which citations to include and how they will inform the introduction and discussion sections. The “Instructions to Authors” for the targeted journal will often provide guidance on structuring the literature review (or introduction) and the number of total citations permitted for each article category. Reviewing articles of similar type published in the targeted journal can also provide guidance regarding structure and average lengths of the introduction and discussion sections.
When selecting references for the introduction consider those that illustrate core background theoretical and methodological concepts, as well as recent relevant studies. The introduction should be brief and present references not as a laundry list or narrative of available literature, but rather as a synthesized summary to provide context for the current study and to identify the gap in the literature that the study intends to fill. For the discussion, citations should be thoughtfully selected to compare and contrast the present study's findings with the current literature and to indicate how the present study moves the field forward.
To facilitate writing a literature review, journals are increasingly providing helpful features to guide authors. For example, the resources available through JGME include several articles on writing. 17 The journal Perspectives on Medical Education recently launched “The Writer's Craft,” which is intended to help medical educators improve their writing. Additionally, many institutions have writing centers that provide web-based materials on writing a literature review, and some even have writing coaches.
The literature review is a vital part of medical education research and should occur throughout the research process to help researchers design a strong study and effectively communicate study results and importance. To achieve these goals, researchers are advised to plan and execute the literature review carefully. The guidance in this editorial provides considerations and recommendations that may improve the quality of literature reviews.
The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Uniformed Services University of the Health Sciences, the Department of the Navy, the Department of Defense, or the US government.
Recipient(s) will receive an email with a link to 'The Literature Review: A Foundation for High-Quality Medical Education Research' and will not need an account to access the content.
Subject: The Literature Review: A Foundation for High-Quality Medical Education Research
(Optional message may have a maximum of 1000 characters.)
Citing articles via
Never miss an issue, get new journal tables of contents sent right to your email inbox.
- Submit an Article
- eISSN 1949-8357
- ISSN 1949-8349
- Get Adobe Acrobat Reader
This Feature Is Available To Subscribers Only
Sign In or Create an Account