significance of the study in research about online learning

• Research article
• Open access
• Published: 02 December 2020

Integrating students’ perspectives about online learning: a hierarchy of factors

• Montgomery Van Wart 1 ,
• Anna Ni 1 ,
• Pamela Medina 1 ,
• Jesus Canelon 1 ,
• Melika Kordrostami 1 ,
• Jing Zhang 1 &

International Journal of Educational Technology in Higher Education volume  17 , Article number:  53 ( 2020 ) Cite this article

150k Accesses

53 Citations

24 Altmetric

Metrics details

This article reports on a large-scale ( n  = 987), exploratory factor analysis study incorporating various concepts identified in the literature as critical success factors for online learning from the students’ perspective, and then determines their hierarchical significance. Seven factors--Basic Online Modality, Instructional Support, Teaching Presence, Cognitive Presence, Online Social Comfort, Online Interactive Modality, and Social Presence--were identified as significant and reliable. Regression analysis indicates the minimal factors for enrollment in future classes—when students consider convenience and scheduling—were Basic Online Modality, Cognitive Presence, and Online Social Comfort. Students who accepted or embraced online courses on their own merits wanted a minimum of Basic Online Modality, Teaching Presence, Cognitive Presence, Online Social Comfort, and Social Presence. Students, who preferred face-to-face classes and demanded a comparable experience, valued Online Interactive Modality and Instructional Support more highly. Recommendations for online course design, policy, and future research are provided.

Introduction

While there are different perspectives of the learning process such as learning achievement and faculty perspectives, students’ perspectives are especially critical since they are ultimately the raison d’être of the educational endeavor (Chickering & Gamson, 1987 ). More pragmatically, students’ perspectives provide invaluable, first-hand insights into their experiences and expectations (Dawson et al., 2019 ). The student perspective is especially important when new teaching approaches are used and when new technologies are being introduced (Arthur, 2009 ; Crews & Butterfield, 2014 ; Van Wart, Ni, Ready, Shayo, & Court, 2020 ). With the renewed interest in “active” education in general (Arruabarrena, Sánchez, Blanco, et al., 2019 ; Kay, MacDonald, & DiGiuseppe, 2019 ; Nouri, 2016 ; Vlachopoulos & Makri, 2017 ) and the flipped classroom approach in particular (Flores, del-Arco, & Silva, 2016 ; Gong, Yang, & Cai, 2020 ; Lundin, et al., 2018 ; Maycock, 2019 ; McGivney-Burelle, 2013 ; O’Flaherty & Phillips, 2015 ; Tucker , 2012 ) along with extraordinary shifts in the technology, the student perspective on online education is profoundly important. What shapes students’ perceptions of quality integrate are their own sense of learning achievement, satisfaction with the support they receive, technical proficiency of the process, intellectual and emotional stimulation, comfort with the process, and sense of learning community. The factors that students perceive as quality online teaching, however, has not been as clear as it might be for at least two reasons.

First, it is important to note that the overall online learning experience for students is also composed of non-teaching factors which we briefly mention. Three such factors are (1) convenience, (2) learner characteristics and readiness, and (3) antecedent conditions that may foster teaching quality but are not directly responsible for it. (1) Convenience is an enormous non-quality factor for students (Artino, 2010 ) which has driven up online demand around the world (Fidalgo, Thormann, Kulyk, et al., 2020 ; Inside Higher Education and Gallup, 2019 ; Legon & Garrett, 2019 ; Ortagus, 2017 ). This is important since satisfaction with online classes is frequently somewhat lower than face-to-face classes (Macon, 2011 ). However, the literature generally supports the relative equivalence of face-to-face and online modes regarding learning achievement criteria (Bernard et al., 2004 ; Nguyen, 2015 ; Ni, 2013 ; Sitzmann, Kraiger, Stewart, & Wisher, 2006 ; see Xu & Jaggars, 2014 for an alternate perspective). These contrasts are exemplified in a recent study of business students, in which online students using a flipped classroom approach outperformed their face-to-face peers, but ironically rated instructor performance lower (Harjoto, 2017 ). (2) Learner characteristics also affect the experience related to self-regulation in an active learning model, comfort with technology, and age, among others,which affect both receptiveness and readiness of online instruction. (Alqurashi, 2016 ; Cohen & Baruth, 2017 ; Kintu, Zhu, & Kagambe, 2017 ; Kuo, Walker, Schroder, & Belland, 2013 ; Ventura & Moscoloni, 2015 ) (3) Finally, numerous antecedent factors may lead to improved instruction, but are not themselves directly perceived by students such as instructor training (Brinkley-Etzkorn, 2018 ), and the sources of faculty motivation (e.g., incentives, recognition, social influence, and voluntariness) (Wingo, Ivankova, & Moss, 2017 ). Important as these factors are, mixing them with the perceptions of quality tends to obfuscate the quality factors directly perceived by students.

Second, while student perceptions of quality are used in innumerable studies, our overall understanding still needs to integrate them more holistically. Many studies use student perceptions of quality and overall effectiveness of individual tools and strategies in online contexts such as mobile devices (Drew & Mann, 2018 ), small groups (Choi, Land, & Turgeon, 2005 ), journals (Nair, Tay, & Koh, 2013 ), simulations (Vlachopoulos & Makri, 2017 ), video (Lange & Costley, 2020 ), etc. Such studies, however, cannot provide the overall context and comparative importance. Some studies have examined the overall learning experience of students with exploratory lists, but have mixed non-quality factors with quality of teaching factors making it difficult to discern the instructor’s versus contextual roles in quality (e.g., Asoodar, Vaezi, & Izanloo, 2016 ; Bollinger & Martindale, 2004 ; Farrell & Brunton, 2020 ; Hong, 2002 ; Song, Singleton, Hill, & Koh, 2004 ; Sun, Tsai, Finger, Chen, & Yeh, 2008 ). The application of technology adoption studies also fall into this category by essentially aggregating all teaching quality in the single category of performance ( Al-Gahtani, 2016 ; Artino, 2010 ). Some studies have used high-level teaching-oriented models, primarily the Community of Inquiry model (le Roux & Nagel, 2018 ), but empirical support has been mixed (Arbaugh et al., 2008 ); and its elegance (i.e., relying on only three factors) has not provided much insight to practitioners (Anderson, 2016 ; Cleveland-Innes & Campbell, 2012 ).

Research questions

Integration of studies and concepts explored continues to be fragmented and confusing despite the fact that the number of empirical studies related to student perceptions of quality factors has increased. It is important to have an empirical view of what students’ value in a single comprehensive study and, also, to know if there is a hierarchy of factors, ranging from students who are least to most critical of the online learning experience. This research study has two research questions.

The first research question is: What are the significant factors in creating a high-quality online learning experience from students’ perspectives? That is important to know because it should have a significant effect on the instructor’s design of online classes. The goal of this research question is identify a more articulated and empirically-supported set of factors capturing the full range of student expectations.

The second research question is: Is there a priority or hierarchy of factors related to students’ perceptions of online teaching quality that relate to their decisions to enroll in online classes? For example, is it possible to distinguish which factors are critical for enrollment decisions when students are primarily motivated by convenience and scheduling flexibility (minimum threshold)? Do these factors differ from students with a genuine acceptance of the general quality of online courses (a moderate threshold)? What are the factors that are important for the students who are the most critical of online course delivery (highest threshold)?

This article next reviews the literature on online education quality, focusing on the student perspective and reviews eight factors derived from it. The research methods section discusses the study structure and methods. Demographic data related to the sample are next, followed by the results, discussion, and conclusion.

Literature review

Online education is much discussed (Prinsloo, 2016 ; Van Wart et al., 2019 ; Zawacki-Richter & Naidu, 2016 ), but its perception is substantially influenced by where you stand and what you value (Otter et al., 2013 ; Tanner, Noser, & Totaro, 2009 ). Accrediting bodies care about meeting technical standards, proof of effectiveness, and consistency (Grandzol & Grandzol, 2006 ). Institutions care about reputation, rigor, student satisfaction, and institutional efficiency (Jung, 2011 ). Faculty care about subject coverage, student participation, faculty satisfaction, and faculty workload (Horvitz, Beach, Anderson, & Xia, 2015 ; Mansbach & Austin, 2018 ). For their part, students care about learning achievement (Marks, Sibley, & Arbaugh, 2005 ; O’Neill & Sai, 2014 ; Shen, Cho, Tsai, & Marra, 2013 ), but also view online education as a function of their enjoyment of classes, instructor capability and responsiveness, and comfort in the learning environment (e.g., Asoodar et al., 2016 ; Sebastianelli, Swift, & Tamimi, 2015 ). It is this last perspective, of students, upon which we focus.

It is important to note students do not sign up for online classes solely based on perceived quality. Perceptions of quality derive from notions of the capacity of online learning when ideal—relative to both learning achievement and satisfaction/enjoyment, and perceptions about the likelihood and experience of classes living up to expectations. Students also sign up because of convenience and flexibility, and personal notions of suitability about learning. Convenience and flexibility are enormous drivers of online registration (Lee, Stringer, & Du, 2017 ; Mann & Henneberry, 2012 ). Even when students say they prefer face-to-face classes to online, many enroll in online classes and re-enroll in the future if the experience meets minimum expectations. This study examines the threshold expectations of students when they are considering taking online classes.

When discussing students’ perceptions of quality, there is little clarity about the actual range of concepts because no integrated empirical studies exist comparing major factors found throughout the literature. Rather, there are practitioner-generated lists of micro-competencies such as the Quality Matters consortium for higher education (Quality Matters, 2018 ), or broad frameworks encompassing many aspects of quality beyond teaching (Open and Distant Learning Quality Council, 2012 ). While checklists are useful for practitioners and accreditation processes, they do not provide robust, theoretical bases for scholarly development. Overarching frameworks are heuristically useful, but not for pragmatic purposes or theory building arenas. The most prominent theoretical framework used in online literature is the Community of Inquiry (CoI) model (Arbaugh et al., 2008 ; Garrison, Anderson, & Archer, 2003 ), which divides instruction into teaching, cognitive, and social presence. Like deductive theories, however, the supportive evidence is mixed (Rourke & Kanuka, 2009 ), especially regarding the importance of social presence (Annand, 2011 ; Armellini and De Stefani, 2016 ). Conceptually, the problem is not so much with the narrow articulation of cognitive or social presence; cognitive presence is how the instructor provides opportunities for students to interact with material in robust, thought-provoking ways, and social presence refers to building a community of learning that incorporates student-to-student interactions. However, teaching presence includes everything else the instructor does—structuring the course, providing lectures, explaining assignments, creating rehearsal opportunities, supplying tests, grading, answering questions, and so on. These challenges become even more prominent in the online context. While the lecture as a single medium is paramount in face-to-face classes, it fades as the primary vehicle in online classes with increased use of detailed syllabi, electronic announcements, recorded and synchronous lectures, 24/7 communications related to student questions, etc. Amassing the pedagogical and technological elements related to teaching under a single concept provides little insight.

In addition to the CoI model, numerous concepts are suggested in single-factor empirical studies when focusing on quality from a student’s perspective, with overlapping conceptualizations and nonstandardized naming conventions. Seven distinct factors are derived here from the literature of student perceptions of online quality: Instructional Support, Teaching Presence, Basic Online Modality, Social Presence, Online Social Comfort, cognitive Presence, and Interactive Online Modality.

Instructional support

Instructional Support refers to students’ perceptions of techniques by the instructor used for input, rehearsal, feedback, and evaluation. Specifically, this entails providing detailed instructions, designed use of multimedia, and the balance between repetitive class features for ease of use, and techniques to prevent boredom. Instructional Support is often included as an element of Teaching Presence, but is also labeled “structure” (Lee & Rha, 2009 ; So & Brush, 2008 ) and instructor facilitation (Eom, Wen, & Ashill, 2006 ). A prime example of the difference between face-to-face and online education is the extensive use of the “flipped classroom” (Maycock, 2019 ; Wang, Huang, & Schunn, 2019 ) in which students move to rehearsal activities faster and more frequently than traditional classrooms, with less instructor lecture (Jung, 2011 ; Martin, Wang, & Sadaf, 2018 ). It has been consistently supported as an element of student perceptions of quality (Espasa & Meneses, 2010 ).

• Teaching presence

Teaching Presence refers to students’ perceptions about the quality of communication in lectures, directions, and individual feedback including encouragement (Jaggars & Xu, 2016 ; Marks et al., 2005 ). Specifically, instructor communication is clear, focused, and encouraging, and instructor feedback is customized and timely. If Instructional Support is what an instructor does before the course begins and in carrying out those plans, then Teaching Presence is what the instructor does while the class is conducted and in response to specific circumstances. For example, a course could be well designed but poorly delivered because the instructor is distracted; or a course could be poorly designed but an instructor might make up for the deficit by spending time and energy in elaborate communications and ad hoc teaching techniques. It is especially important in student satisfaction (Sebastianelli et al., 2015 ; Young, 2006 ) and also referred to as instructor presence (Asoodar et al., 2016 ), learner-instructor interaction (Marks et al., 2005 ), and staff support (Jung, 2011 ). As with Instructional Support, it has been consistently supported as an element of student perceptions of quality.

Basic online modality

Basic Online Modality refers to the competent use of basic online class tools—online grading, navigation methods, online grade book, and the announcements function. It is frequently clumped with instructional quality (Artino, 2010 ), service quality (Mohammadi, 2015 ), instructor expertise in e-teaching (Paechter, Maier, & Macher, 2010 ), and similar terms. As a narrowly defined concept, it is sometimes called technology (Asoodar et al., 2016 ; Bollinger & Martindale, 2004 ; Sun et al., 2008 ). The only empirical study that did not find Basic Online Modality significant, as technology, was Sun et al. ( 2008 ). Because Basic Online Modality is addressed with basic instructor training, some studies assert the importance of training (e.g., Asoodar et al., 2016 ).

Social presence

Social Presence refers to students’ perceptions of the quality of student-to-student interaction. Social Presence focuses on the quality of shared learning and collaboration among students, such as in threaded discussion responses (Garrison et al., 2003 ; Kehrwald, 2008 ). Much emphasized but challenged in the CoI literature (Rourke & Kanuka, 2009 ), it has mixed support in the online literature. While some studies found Social Presence or related concepts to be significant (e.g., Asoodar et al., 2016 ; Bollinger & Martindale, 2004 ; Eom et al., 2006 ; Richardson, Maeda, Lv, & Caskurlu, 2017 ), others found Social Presence insignificant (Joo, Lim, & Kim, 2011 ; So & Brush, 2008 ; Sun et al., 2008 ).

Online social comfort

Online Social Comfort refers to the instructor’s ability to provide an environment in which anxiety is low, and students feel comfortable interacting even when expressing opposing viewpoints. While numerous studies have examined anxiety (e.g., Liaw & Huang, 2013 ; Otter et al., 2013 ; Sun et al., 2008 ), only one found anxiety insignificant (Asoodar et al., 2016 ); many others have not examined the concept.

• Cognitive presence

Cognitive Presence refers to the engagement of students such that they perceive they are stimulated by the material and instructor to reflect deeply and critically, and seek to understand different perspectives (Garrison et al., 2003 ). The instructor provides instructional materials and facilitates an environment that piques interest, is reflective, and enhances inclusiveness of perspectives (Durabi, Arrastia, Nelson, Cornille, & Liang, 2011 ). Cognitive Presence includes enhancing the applicability of material for student’s potential or current careers. Cognitive Presence is supported as significant in many online studies (e.g., Artino, 2010 ; Asoodar et al., 2016 ; Joo et al., 2011 ; Marks et al., 2005 ; Sebastianelli et al., 2015 ; Sun et al., 2008 ). Further, while many instructors perceive that cognitive presence is diminished in online settings, neuroscientific studies indicate this need not be the case (Takamine, 2017 ). While numerous studies failed to examine Cognitive Presence, this review found no studies that lessened its significance for students.

Interactive online modality

Interactive Online Modality refers to the “high-end” usage of online functionality. That is, the instructor uses interactive online class tools—video lectures, videoconferencing, and small group discussions—well. It is often included in concepts such as instructional quality (Artino, 2010 ; Asoodar et al., 2016 ; Mohammadi, 2015 ; Otter et al., 2013 ; Paechter et al., 2010 ) or engagement (Clayton, Blumberg, & Anthony, 2018 ). While individual methods have been investigated (e.g. Durabi et al., 2011 ), high-end engagement methods have not.

Other independent variables affecting perceptions of quality include age, undergraduate versus graduate status, gender, ethnicity/race, discipline, educational motivation of students, and previous online experience. While age has been found to be small or insignificant, more notable effects have been reported at the level-of-study, with graduate students reporting higher “success” (Macon, 2011 ), and community college students having greater difficulty with online classes (Legon & Garrett, 2019 ; Xu & Jaggars, 2014 ). Ethnicity and race have also been small or insignificant. Some situational variations and student preferences can be captured by paying attention to disciplinary differences (Arbaugh, 2005 ; Macon, 2011 ). Motivation levels of students have been reported to be significant in completion and achievement, with better students doing as well across face-to-face and online modes, and weaker students having greater completion and achievement challenges (Clayton et al., 2018 ; Lu & Lemonde, 2013 ).

Research methods

To examine the various quality factors, we apply a critical success factor methodology, initially introduced to schools of business research in the 1970s. In 1981, Rockhart and Bullen codified an approach embodying principles of critical success factors (CSFs) as a way to identify the information needs of executives, detailing steps for the collection and analyzation of data to create a set of organizational CSFs (Rockhart & Bullen, 1981 ). CSFs describe the underlying or guiding principles which must be incorporated to ensure success.

Utilizing this methodology, CSFs in the context of this paper define key areas of instruction and design essential for an online class to be successful from a student’s perspective. Instructors implicitly know and consider these areas when setting up an online class and designing and directing activities and tasks important to achieving learning goals. CSFs make explicit those things good instructors may intuitively know and (should) do to enhance student learning. When made explicit, CSFs not only confirm the knowledge of successful instructors, but tap their intuition to guide and direct the accomplishment of quality instruction for entire programs. In addition, CSFs are linked with goals and objectives, helping generate a small number of truly important matters an instructor should focus attention on to achieve different thresholds of online success.

After a comprehensive literature review, an instrument was created to measure students’ perceptions about the importance of techniques and indicators leading to quality online classes. Items were designed to capture the major factors in the literature. The instrument was pilot studied during academic year 2017–18 with a 397 student sample, facilitating an exploratory factor analysis leading to important preliminary findings (reference withheld for review). Based on the pilot, survey items were added and refined to include seven groups of quality teaching factors and two groups of items related to students’ overall acceptance of online classes as well as a variable on their future online class enrollment. Demographic information was gathered to determine their effects on students’ levels of acceptance of online classes based on age, year in program, major, distance from university, number of online classes taken, high school experience with online classes, and communication preferences.

This paper draws evidence from a sample of students enrolled in educational programs at Jack H. Brown College of Business and Public Administration (JHBC), California State University San Bernardino (CSUSB). The JHBC offers a wide range of online courses for undergraduate and graduate programs. To ensure comparable learning outcomes, online classes and face-to-face classes of a certain subject are similar in size—undergraduate classes are generally capped at 60 and graduate classes at 30, and often taught by the same instructors. Students sometimes have the option to choose between both face-to-face and online modes of learning.

A Qualtrics survey link was sent out by 11 instructors to students who were unlikely to be cross-enrolled in classes during the 2018–19 academic year. 1 Approximately 2500 students were contacted, with some instructors providing class time to complete the anonymous survey. All students, whether they had taken an online class or not, were encouraged to respond. Nine hundred eighty-seven students responded, representing a 40% response rate. Although drawn from a single business school, it is a broad sample representing students from several disciplines—management, accounting and finance, marketing, information decision sciences, and public administration, as well as both graduate and undergraduate programs of study.

The sample age of students is young, with 78% being under 30. The sample has almost no lower division students (i.e., freshman and sophomore), 73% upper division students (i.e., junior and senior) and 24% graduate students (master’s level). Only 17% reported having taken a hybrid or online class in high school. There was a wide range of exposure to university level online courses, with 47% reporting having taken 1 to 4 classes, and 21% reporting no online class experience. As a Hispanic-serving institution, 54% self-identified as Latino, 18% White, and 13% Asian and Pacific Islander. The five largest majors were accounting & finance (25%), management (21%), master of public administration (16%), marketing (12%), and information decision sciences (10%). Seventy-four percent work full- or part-time. See Table  1 for demographic data.

Measures and procedure

To increase the reliability of evaluation scores, composite evaluation variables are formed after an exploratory factor analysis of individual evaluation items. A principle component method with Quartimin (oblique) rotation was applied to explore the factor construct of student perceptions of online teaching CSFs. The item correlations for student perceptions of importance coefficients greater than .30 were included, a commonly acceptable ratio in factor analysis. A simple least-squares regression analysis was applied to test the significance levels of factors on students’ impression of online classes.

Exploratory factor constructs

Using a threshold loading of 0.3 for items, 37 items loaded on seven factors. All factors were logically consistent. The first factor, with eight items, was labeled Teaching Presence. Items included providing clear instructions, staying on task, clear deadlines, and customized feedback on strengths and weaknesses. Teaching Presence items all related to instructor involvement during the course as a director, monitor, and learning facilitator. The second factor, with seven items, aligned with Cognitive Presence. Items included stimulating curiosity, opportunities for reflection, helping students construct explanations posed in online courses, and the applicability of material. The third factor, with six items, aligned with Social Presence defined as providing student-to-student learning opportunities. Items included getting to know course participants for sense of belonging, forming impressions of other students, and interacting with others. The fourth factor, with six new items as well as two (“interaction with other students” and “a sense of community in the class”) shared with the third factor, was Instructional Support which related to the instructor’s roles in providing students a cohesive learning experience. They included providing sufficient rehearsal, structured feedback, techniques for communication, navigation guide, detailed syllabus, and coordinating student interaction and creating a sense of online community. This factor also included enthusiasm which students generally interpreted as a robustly designed course, rather than animation in a traditional lecture. The fifth factor was labeled Basic Online Modality and focused on the basic technological requirements for a functional online course. Three items included allowing students to make online submissions, use of online gradebooks, and online grading. A fourth item is the use of online quizzes, viewed by students as mechanical practice opportunities rather than small tests and a fifth is navigation, a key component of Online Modality. The sixth factor, loaded on four items, was labeled Online Social Comfort. Items here included comfort discussing ideas online, comfort disagreeing, developing a sense of collaboration via discussion, and considering online communication as an excellent medium for social interaction. The final factor was called Interactive Online Modality because it included items for “richer” communications or interactions, no matter whether one- or two-way. Items included videoconferencing, instructor-generated videos, and small group discussions. Taken together, these seven explained 67% of the variance which is considered in the acceptable range in social science research for a robust model (Hair, Black, Babin, & Anderson, 2014 ). See Table  2 for the full list.

To test for factor reliability, the Cronbach alpha of variables were calculated. All produced values greater than 0.7, the standard threshold used for reliability, except for system trust which was therefore dropped. To gauge students’ sense of factor importance, all items were means averaged. Factor means (lower means indicating higher importance to students), ranged from 1.5 to 2.6 on a 5-point scale. Basic Online Modality was most important, followed by Instructional Support and Teaching Presence. Students deemed Cognitive Presence, Social Online Comfort, and Online Interactive Modality less important. The least important for this sample was Social Presence. Table  3 arrays the critical success factor means, standard deviations, and Cronbach alpha.

To determine whether particular subgroups of respondents viewed factors differently, a series of ANOVAs were conducted using factor means as dependent variables. Six demographic variables were used as independent variables: graduate vs. undergraduate, age, work status, ethnicity, discipline, and past online experience. To determine strength of association of the independent variables to each of the seven CSFs, eta squared was calculated for each ANOVA. Eta squared indicates the proportion of variance in the dependent variable explained by the independent variable. Eta squared values greater than .01, .06, and .14 are conventionally interpreted as small, medium, and large effect sizes, respectively (Green & Salkind, 2003 ). Table  4 summarizes the eta squared values for the ANOVA tests with Eta squared values less than .01 omitted.

While no significant differences in factor means among students in different disciplines in the College occur, all five other independent variables have some small effect on some or all CSFs. Graduate students tend to rate Online Interactive Modality, Instructional Support, Teaching Presence, and Cognitive Presence higher than undergraduates. Elder students value more Online Interactive Modality. Full-time working students rate all factors, except Social Online Comfort, slightly higher than part-timers and non-working students. Latino and White rate Basic Online Modality and Instructional Support higher; Asian and Pacific Islanders rate Social Presence higher. Students who have taken more online classes rate all factors higher.

In addition to factor scores, two variables are constructed to identify the resultant impressions labeled online experience. Both were logically consistent with a Cronbach’s α greater than 0.75. The first variable, with six items, labeled “online acceptance,” included items such as “I enjoy online learning,” “My overall impression of hybrid/online learning is very good,” and “the instructors of online/hybrid classes are generally responsive.” The second variable was labeled “face-to-face preference” and combines four items, including enjoying, learning, and communicating more in face-to-face classes, as well as perceiving greater fairness and equity. In addition to these two constructed variables, a one-item variable was also used subsequently in the regression analysis: “online enrollment.” That question asked: if hybrid/online classes are well taught and available, how much would online education make up your entire course selection going forward?

Regression results

As noted above, two constructed variables and one item were used as dependent variables for purposes of regression analysis. They were online acceptance, F2F preference, and the selection of online classes. In addition to seven quality-of-teaching factors identified by factor analysis, control variables included level of education (graduate versus undergraduate), age, ethnicity, work status, distance to university, and number of online/hybrid classes taken in the past. See Table  5 .

When the ETA squared values for ANOVA significance were measured for control factors, only one was close to a medium effect. Graduate versus undergraduate status had a .05 effect (considered medium) related to Online Interactive Modality, meaning graduate students were more sensitive to interactive modality than undergraduates. Multiple regression analysis of critical success factors and online impressions were conducted to compare under what conditions factors were significant. The only consistently significant control factor was number of online classes taken. The more classes students had taken online, the more inclined they were to take future classes. Level of program, age, ethnicity, and working status do not significantly affect students’ choice or overall acceptance of online classes.

The least restrictive condition was online enrollment (Table  6 ). That is, students might not feel online courses were ideal, but because of convenience and scheduling might enroll in them if minimum threshold expectations were met. When considering online enrollment three factors were significant and positive (at the 0.1 level): Basic Online Modality, Cognitive Presence, and Online Social Comfort. These least-demanding students expected classes to have basic technological functionality, provide good opportunities for knowledge acquisition, and provide comfortable interaction in small groups. Students who demand good Instructional Support (e.g., rehearsal opportunities, standardized feedback, clear syllabus) are less likely to enroll.

Online acceptance was more restrictive (see Table  7 ). This variable captured the idea that students not only enrolled in online classes out of necessity, but with an appreciation of the positive attributes of online instruction, which balanced the negative aspects. When this standard was applied, students expected not only Basic Online Modality, Cognitive Presence, and Online Social Comfort, but expected their instructors to be highly engaged virtually as the course progressed (Teaching Presence), and to create strong student-to-student dynamics (Social Presence). Students who rated Instructional Support higher are less accepting of online classes.

Another restrictive condition was catering to the needs of students who preferred face-to-face classes (see Table  8 ). That is, they preferred face-to-face classes even when online classes were well taught. Unlike students more accepting of, or more likely to enroll in, online classes, this group rates Instructional Support as critical to enrolling, rather than a negative factor when absent. Again different from the other two groups, these students demand appropriate interactive mechanisms (Online Interactive Modality) to enable richer communication (e.g., videoconferencing). Student-to-student collaboration (Social Presence) was also significant. This group also rated Cognitive Presence and Online Social Comfort as significant, but only in their absence. That is, these students were most attached to direct interaction with the instructor and other students rather than specific teaching methods. Interestingly, Basic Online Modality and Teaching Presence were not significant. Our interpretation here is this student group, most critical of online classes for its loss of physical interaction, are beyond being concerned with mechanical technical interaction and demand higher levels of interactivity and instructional sophistication.

Discussion and study limitations

Some past studies have used robust empirical methods to identify a single factor or a small number of factors related to quality from a student’s perspective, but have not sought to be relatively comprehensive. Others have used a longer series of itemized factors, but have less used less robust methods, and have not tied those factors back to the literature. This study has used the literature to develop a relatively comprehensive list of items focused on quality teaching in a single rigorous protocol. That is, while a Beta test had identified five coherent factors, substantial changes to the current survey that sharpened the focus on quality factors rather than antecedent factors, as well as better articulating the array of factors often lumped under the mantle of “teaching presence.” In addition, it has also examined them based on threshold expectations: from minimal, such as when flexibility is the driving consideration, to modest, such as when students want a “good” online class, to high, when students demand an interactive virtual experience equivalent to face-to-face.

Exploratory factor analysis identified seven factors that were reliable, coherent, and significant under different conditions. When considering students’ overall sense of importance, they are, in order: Basic Online Modality, Instructional Support, Teaching Presence, Cognitive Presence, Social Online Comfort, Interactive Online Modality, and Social Presence. Students are most concerned with the basics of a course first, that is the technological and instructor competence. Next they want engagement and virtual comfort. Social Presence, while valued, is the least critical from this overall perspective.

The factor analysis is quite consistent with the range of factors identified in the literature, pointing to the fact that students can differentiate among different aspects of what have been clumped as larger concepts, such as teaching presence. Essentially, the instructor’s role in quality can be divided into her/his command of basic online functionality, good design, and good presence during the class. The instructor’s command of basic functionality is paramount. Because so much of online classes must be built in advance of the class, quality of the class design is rated more highly than the instructor’s role in facilitating the class. Taken as a whole, the instructor’s role in traditional teaching elements is primary, as we would expect it to be. Cognitive presence, especially as pertinence of the instructional material and its applicability to student interests, has always been found significant when studied, and was highly rated as well in a single factor. Finally, the degree to which students feel comfortable with the online environment and enjoy the learner-learner aspect has been less supported in empirical studies, was found significant here, but rated the lowest among the factors of quality to students.

Regression analysis paints a more nuanced picture, depending on student focus. It also helps explain some of the heterogeneity of previous studies, depending on what the dependent variables were. If convenience and scheduling are critical and students are less demanding, minimum requirements are Basic Online Modality, Cognitive Presence, and Online Social Comfort. That is, students’ expect an instructor who knows how to use an online platform, delivers useful information, and who provides a comfortable learning environment. However, they do not expect to get poor design. They do not expect much in terms of the quality teaching presence, learner-to-learner interaction, or interactive teaching.

When students are signing up for critical classes, or they have both F2F and online options, they have a higher standard. That is, they not only expect the factors for decisions about enrolling in noncritical classes, but they also expect good Teaching and Social Presence. Students who simply need a class may be willing to teach themselves a bit more, but students who want a good class expect a highly present instructor in terms responsiveness and immediacy. “Good” classes must not only create a comfortable atmosphere, but in social science classes at least, must provide strong learner-to-learner interactions as well. At the time of the research, most students believe that you can have a good class without high interactivity via pre-recorded video and videoconference. That may, or may not, change over time as technology thresholds of various video media become easier to use, more reliable, and more commonplace.

The most demanding students are those who prefer F2F classes because of learning style preferences, poor past experiences, or both. Such students (seem to) assume that a worthwhile online class has basic functionality and that the instructor provides a strong presence. They are also critical of the absence of Cognitive Presence and Online Social Comfort. They want strong Instructional Support and Social Presence. But in addition, and uniquely, they expect Online Interactive Modality which provides the greatest verisimilitude to the traditional classroom as possible. More than the other two groups, these students crave human interaction in the learning process, both with the instructor and other students.

These findings shed light on the possible ramifications of the COVID-19 aftermath. Many universities around the world jumped from relatively low levels of online instruction in the beginning of spring 2020 to nearly 100% by mandate by the end of the spring term. The question becomes, what will happen after the mandate is removed? Will demand resume pre-crisis levels, will it increase modestly, or will it skyrocket? Time will be the best judge, but the findings here would suggest that the ability/interest of instructors and institutions to “rise to the occasion” with quality teaching will have as much effect on demand as students becoming more acclimated to online learning. If in the rush to get classes online many students experience shoddy basic functional competence, poor instructional design, sporadic teaching presence, and poorly implemented cognitive and social aspects, they may be quite willing to return to the traditional classroom. If faculty and institutions supporting them are able to increase the quality of classes despite time pressures, then most students may be interested in more hybrid and fully online classes. If instructors are able to introduce high quality interactive teaching, nearly the entire student population will be interested in more online classes. Of course students will have a variety of experiences, but this analysis suggests that those instructors, departments, and institutions that put greater effort into the temporary adjustment (and who resist less), will be substantially more likely to have increases in demand beyond what the modest national trajectory has been for the last decade or so.

There are several study limitations. First, the study does not include a sample of non-respondents. Non-responders may have a somewhat different profile. Second, the study draws from a single college and university. The profile derived here may vary significantly by type of student. Third, some survey statements may have led respondents to rate quality based upon experience rather than assess the general importance of online course elements. “I felt comfortable participating in the course discussions,” could be revised to “comfort in participating in course discussions.” The authors weighed differences among subgroups (e.g., among majors) as small and statistically insignificant. However, it is possible differences between biology and marketing students would be significant, leading factors to be differently ordered. Emphasis and ordering might vary at a community college versus research-oriented university (Gonzalez, 2009 ).

Availability of data and materials

We will make the data available.

Al-Gahtani, S. S. (2016). Empirical investigation of e-learning acceptance and assimilation: A structural equation model. Applied Comput Information , 12 , 27–50.

Google Scholar  

Alqurashi, E. (2016). Self-efficacy in online learning environments: A literature review. Contemporary Issues Educ Res (CIER) , 9 (1), 45–52.

Anderson, T. (2016). A fourth presence for the Community of Inquiry model? Retrieved from https://virtualcanuck.ca/2016/01/04/a-fourth-presence-for-the-community-of-inquiry-model/ .

Annand, D. (2011). Social presence within the community of inquiry framework. The International Review of Research in Open and Distributed Learning , 12 (5), 40.

Arbaugh, J. B. (2005). How much does “subject matter” matter? A study of disciplinary effects in on-line MBA courses. Academy of Management Learning & Education , 4 (1), 57–73.

Arbaugh, J. B., Cleveland-Innes, M., Diaz, S. R., Garrison, D. R., Ice, P., Richardson, J. C., & Swan, K. P. (2008). Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry framework using a multi-institutional sample. Internet and Higher Education , 11 , 133–136.

Armellini, A., & De Stefani, M. (2016). Social presence in the 21st century: An adjustment to the Community of Inquiry framework. British Journal of Educational Technology , 47 (6), 1202–1216.

Arruabarrena, R., Sánchez, A., Blanco, J. M., et al. (2019). Integration of good practices of active methodologies with the reuse of student-generated content. International Journal of Educational Technology in Higher Education , 16 , #10.

Arthur, L. (2009). From performativity to professionalism: Lecturers’ responses to student feedback. Teaching in Higher Education , 14 (4), 441–454.

Artino, A. R. (2010). Online or face-to-face learning? Exploring the personal factors that predict students’ choice of instructional format. Internet and Higher Education , 13 , 272–276.

Asoodar, M., Vaezi, S., & Izanloo, B. (2016). Framework to improve e-learner satisfaction and further strengthen e-learning implementation. Computers in Human Behavior , 63 , 704–716.

Bernard, R. M., et al. (2004). How does distance education compare with classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research , 74 (3), 379–439.

Bollinger, D., & Martindale, T. (2004). Key factors for determining student satisfaction in online courses. Int J E-learning , 3 (1), 61–67.

Brinkley-Etzkorn, K. E. (2018). Learning to teach online: Measuring the influence of faculty development training on teaching effectiveness through a TPACK lens. The Internet and Higher Education , 38 , 28–35.

Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin , 3 , 7.

Choi, I., Land, S. M., & Turgeon, A. J. (2005). Scaffolding peer-questioning strategies to facilitate metacognition during online small group discussion. Instructional Science , 33 , 483–511.

Clayton, K. E., Blumberg, F. C., & Anthony, J. A. (2018). Linkages between course status, perceived course value, and students’ preferences for traditional versus non-traditional learning environments. Computers & Education , 125 , 175–181.

Cleveland-Innes, M., & Campbell, P. (2012). Emotional presence, learning, and the online learning environment. The International Review of Research in Open and Distributed Learning , 13 (4), 269–292.

Cohen, A., & Baruth, O. (2017). Personality, learning, and satisfaction in fully online academic courses. Computers in Human Behavior , 72 , 1–12.

Crews, T., & Butterfield, J. (2014). Data for flipped classroom design: Using student feedback to identify the best components from online and face-to-face classes. Higher Education Studies , 4 (3), 38–47.

Dawson, P., Henderson, M., Mahoney, P., Phillips, M., Ryan, T., Boud, D., & Molloy, E. (2019). What makes for effective feedback: Staff and student perspectives. Assessment & Evaluation in Higher Education , 44 (1), 25–36.

Drew, C., & Mann, A. (2018). Unfitting, uncomfortable, unacademic: A sociological reading of an interactive mobile phone app in university lectures. International Journal of Educational Technology in Higher Education , 15 , #43.

Durabi, A., Arrastia, M., Nelson, D., Cornille, T., & Liang, X. (2011). Cognitive presence in asynchronous online learning: A comparison of four discussion strategies. Journal of Computer Assisted Learning , 27 (3), 216–227.

Eom, S. B., Wen, H. J., & Ashill, N. (2006). The determinants of students’ perceived learning outcomes and satisfaction in university online education: An empirical investigation. Decision Sciences Journal of Innovative Education , 4 (2), 215–235.

Espasa, A., & Meneses, J. (2010). Analysing feedback processes in an online teaching and learning environment: An exploratory study. Higher Education , 59 (3), 277–292.

Farrell, O., & Brunton, J. (2020). A balancing act: A window into online student engagement experiences. International Journal of Educational Technology in High Education , 17 , #25.

Fidalgo, P., Thormann, J., Kulyk, O., et al. (2020). Students’ perceptions on distance education: A multinational study. International Journal of Educational Technology in High Education , 17 , #18.

Flores, Ò., del-Arco, I., & Silva, P. (2016). The flipped classroom model at the university: Analysis based on professors’ and students’ assessment in the educational field. International Journal of Educational Technology in Higher Education , 13 , #21.

Garrison, D. R., Anderson, T., & Archer, W. (2003). A theory of critical inquiry in online distance education. Handbook of Distance Education , 1 , 113–127.

Gong, D., Yang, H. H., & Cai, J. (2020). Exploring the key influencing factors on college students’ computational thinking skills through flipped-classroom instruction. International Journal of Educational Technology in Higher Education , 17 , #19.

Gonzalez, C. (2009). Conceptions of, and approaches to, teaching online: A study of lecturers teaching postgraduate distance courses. Higher Education , 57 (3), 299–314.

Grandzol, J. R., & Grandzol, C. J. (2006). Best practices for online business Education. International Review of Research in Open and Distance Learning , 7 (1), 1–18.

Green, S. B., & Salkind, N. J. (2003). Using SPSS: Analyzing and understanding data , (3rd ed., ). Upper Saddle River: Prentice Hall.

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2014). Multivariate data analysis: Pearson new international edition . Essex: Pearson Education Limited.

Harjoto, M. A. (2017). Blended versus face-to-face: Evidence from a graduate corporate finance class. Journal of Education for Business , 92 (3), 129–137.

Hong, K.-S. (2002). Relationships between students’ instructional variables with satisfaction and learning from a web-based course. The Internet and Higher Education , 5 , 267–281.

Horvitz, B. S., Beach, A. L., Anderson, M. L., & Xia, J. (2015). Examination of faculty self-efficacy related to online teaching. Innovation Higher Education , 40 , 305–316.

Inside Higher Education and Gallup. (2019). The 2019 survey of faculty attitudes on technology. Author .

Jaggars, S. S., & Xu, D. (2016). How do online course design features influence student performance? Computers and Education , 95 , 270–284.

Joo, Y. J., Lim, K. Y., & Kim, E. K. (2011). Online university students’ satisfaction and persistence: Examining perceived level of presence, usefulness and ease of use as predictor in a structural model. Computers & Education , 57 (2), 1654–1664.

Jung, I. (2011). The dimensions of e-learning quality: From the learner’s perspective. Educational Technology Research and Development , 59 (4), 445–464.

Kay, R., MacDonald, T., & DiGiuseppe, M. (2019). A comparison of lecture-based, active, and flipped classroom teaching approaches in higher education. Journal of Computing in Higher Education , 31 , 449–471.

Kehrwald, B. (2008). Understanding social presence in text-based online learning environments. Distance Education , 29 (1), 89–106.

Kintu, M. J., Zhu, C., & Kagambe, E. (2017). Blended learning effectiveness: The relationship between student characteristics, design features and outcomes. International Journal of Educational Technology in Higher Education , 14 , #7.

Kuo, Y.-C., Walker, A. E., Schroder, K. E., & Belland, B. R. (2013). Interaction, internet self-efficacy, and self-regulated learning as predictors of student satisfaction in online education courses. Internet and Education , 20 , 35–50.

Lange, C., & Costley, J. (2020). Improving online video lectures: Learning challenges created by media. International Journal of Educational Technology in Higher Education , 17 , #16.

le Roux, I., & Nagel, L. (2018). Seeking the best blend for deep learning in a flipped classroom – Viewing student perceptions through the Community of Inquiry lens. International Journal of Educational Technology in High Education , 15 , #16.

Lee, H.-J., & Rha, I. (2009). Influence of structure and interaction on student achievement and satisfaction in web-based distance learning. Educational Technology & Society , 12 (4), 372–382.

Lee, Y., Stringer, D., & Du, J. (2017). What determines students’ preference of online to F2F class? Business Education Innovation Journal , 9 (2), 97–102.

Legon, R., & Garrett, R. (2019). CHLOE 3: Behind the numbers . Published online by Quality Matters and Eduventures. https://www.qualitymatters.org/sites/default/files/research-docs-pdfs/CHLOE-3-Report-2019-Behind-the-Numbers.pdf

Liaw, S.-S., & Huang, H.-M. (2013). Perceived satisfaction, perceived usefulness and interactive learning environments as predictors of self-regulation in e-learning environments. Computers & Education , 60 (1), 14–24.

Lu, F., & Lemonde, M. (2013). A comparison of online versus face-to-face students teaching delivery in statistics instruction for undergraduate health science students. Advances in Health Science Education , 18 , 963–973.

Lundin, M., Bergviken Rensfeldt, A., Hillman, T., Lantz-Andersson, A., & Peterson, L. (2018). Higher education dominance and siloed knowledge: a systematic review of flipped classroom research. International Journal of Educational Technology in Higher Education , 15 (1).

Macon, D. K. (2011). Student satisfaction with online courses versus traditional courses: A meta-analysis . Disssertation: Northcentral University, CA.

Mann, J., & Henneberry, S. (2012). What characteristics of college students influence their decisions to select online courses? Online Journal of Distance Learning Administration , 15 (5), 1–14.

Mansbach, J., & Austin, A. E. (2018). Nuanced perspectives about online teaching: Mid-career senior faculty voices reflecting on academic work in the digital age. Innovative Higher Education , 43 (4), 257–272.

Marks, R. B., Sibley, S. D., & Arbaugh, J. B. (2005). A structural equation model of predictors for effective online learning. Journal of Management Education , 29 (4), 531–563.

Martin, F., Wang, C., & Sadaf, A. (2018). Student perception of facilitation strategies that enhance instructor presence, connectedness, engagement and learning in online courses. Internet and Higher Education , 37 , 52–65.

Maycock, K. W. (2019). Chalk and talk versus flipped learning: A case study. Journal of Computer Assisted Learning , 35 , 121–126.

McGivney-Burelle, J. (2013). Flipping Calculus. PRIMUS Problems, Resources, and Issues in Mathematics Undergraduate . Studies , 23 (5), 477–486.

Mohammadi, H. (2015). Investigating users’ perspectives on e-learning: An integration of TAM and IS success model. Computers in Human Behavior , 45 , 359–374.

Nair, S. S., Tay, L. Y., & Koh, J. H. L. (2013). Students’ motivation and teachers’ teaching practices towards the use of blogs for writing of online journals. Educational Media International , 50 (2), 108–119.

Nguyen, T. (2015). The effectiveness of online learning: Beyond no significant difference and future horizons. MERLOT Journal of Online Learning and Teaching , 11 (2), 309–319.

Ni, A. Y. (2013). Comparing the effectiveness of classroom and online learning: Teaching research methods. Journal of Public Affairs Education , 19 (2), 199–215.

Nouri, J. (2016). The flipped classroom: For active, effective and increased learning – Especially for low achievers. International Journal of Educational Technology in Higher Education , 13 , #33.

O’Neill, D. K., & Sai, T. H. (2014). Why not? Examining college students’ reasons for avoiding an online course. Higher Education , 68 (1), 1–14.

O'Flaherty, J., & Phillips, C. (2015). The use of flipped classrooms in higher education: A scoping review. The Internet and Higher Education , 25 , 85–95.

Open & Distant Learning Quality Council (2012). ODLQC standards . England: Author https://www.odlqc.org.uk/odlqc-standards .

Ortagus, J. C. (2017). From the periphery to prominence: An examination of the changing profile of online students in American higher education. Internet and Higher Education , 32 , 47–57.

Otter, R. R., Seipel, S., Graef, T., Alexander, B., Boraiko, C., Gray, J., … Sadler, K. (2013). Comparing student and faculty perceptions of online and traditional courses. Internet and Higher Education , 19 , 27–35.

Paechter, M., Maier, B., & Macher, D. (2010). Online or face-to-face? Students’ experiences and preferences in e-learning. Internet and Higher Education , 13 , 292–329.

Prinsloo, P. (2016). (re)considering distance education: Exploring its relevance, sustainability and value contribution. Distance Education , 37 (2), 139–145.

Quality Matters (2018). Specific review standards from the QM higher Education rubric , (6th ed., ). MD: MarylandOnline.

Richardson, J. C., Maeda, Y., Lv, J., & Caskurlu, S. (2017). Social presence in relation to students’ satisfaction and learning in the online environment: A meta-analysis. Computers in Human Behavior , 71 , 402–417.

Rockhart, J. F., & Bullen, C. V. (1981). A primer on critical success factors . Cambridge: Center for Information Systems Research, Massachusetts Institute of Technology.

Rourke, L., & Kanuka, H. (2009). Learning in Communities of Inquiry: A Review of the Literature. The Journal of Distance Education / Revue de l'ducation Distance , 23 (1), 19–48 Athabasca University Press. Retrieved August 2, 2020 from https://www.learntechlib.org/p/105542/ .

Sebastianelli, R., Swift, C., & Tamimi, N. (2015). Factors affecting perceived learning, satisfaction, and quality in the online MBA: A structural equation modeling approach. Journal of Education for Business , 90 (6), 296–305.

Shen, D., Cho, M.-H., Tsai, C.-L., & Marra, R. (2013). Unpacking online learning experiences: Online learning self-efficacy and learning satisfaction. Internet and Higher Education , 19 , 10–17.

Sitzmann, T., Kraiger, K., Stewart, D., & Wisher, R. (2006). The comparative effectiveness of web-based and classroom instruction: A meta-analysis. Personnel Psychology , 59 (3), 623–664.

So, H. J., & Brush, T. A. (2008). Student perceptions of collaborative learning, social presence and satisfaction in a blended learning environment: Relationships and critical factors. Computers & Education , 51 (1), 318–336.

Song, L., Singleton, E. S., Hill, J. R., & Koh, M. H. (2004). Improving online learning: Student perceptions of useful and challenging characteristics. The Internet and Higher Education , 7 (1), 59–70.

Sun, P. C., Tsai, R. J., Finger, G., Chen, Y. Y., & Yeh, D. (2008). What drives a successful e-learning? An empirical investigation of the critical factors influencing learner satisfaction. Computers & Education , 50 (4), 1183–1202.

Takamine, K. (2017). Michelle D. miller: Minds online: Teaching effectively with technology. Higher Education , 73 , 789–791.

Tanner, J. R., Noser, T. C., & Totaro, M. W. (2009). Business faculty and undergraduate students’ perceptions of online learning: A comparative study. Journal of Information Systems Education , 20 (1), 29.

Tucker, B. (2012). The flipped classroom. Education Next , 12 (1), 82–83.

Van Wart, M., Ni, A., Ready, D., Shayo, C., & Court, J. (2020). Factors leading to online learner satisfaction. Business Educational Innovation Journal , 12 (1), 15–24.

Van Wart, M., Ni, A., Rose, L., McWeeney, T., & Worrell, R. A. (2019). Literature review and model of online teaching effectiveness integrating concerns for learning achievement, student satisfaction, faculty satisfaction, and institutional results. Pan-Pacific . Journal of Business Research , 10 (1), 1–22.

Ventura, A. C., & Moscoloni, N. (2015). Learning styles and disciplinary differences: A cross-sectional study of undergraduate students. International Journal of Learning and Teaching , 1 (2), 88–93.

Vlachopoulos, D., & Makri, A. (2017). The effect of games and simulations on higher education: A systematic literature review. International Journal of Educational Technology in Higher Education , 14 , #22.

Wang, Y., Huang, X., & Schunn, C. D. (2019). Redesigning flipped classrooms: A learning model and its effects on student perceptions. Higher Education , 78 , 711–728.

Wingo, N. P., Ivankova, N. V., & Moss, J. A. (2017). Faculty perceptions about teaching online: Exploring the literature using the technology acceptance model as an organizing framework. Online Learning , 21 (1), 15–35.

Xu, D., & Jaggars, S. S. (2014). Performance gaps between online and face-to-face courses: Differences across types of students and academic subject areas. Journal of Higher Education , 85 (5), 633–659.

Young, S. (2006). Student views of effective online teaching in higher education. American Journal of Distance Education , 20 (2), 65–77.

Zawacki-Richter, O., & Naidu, S. (2016). Mapping research trends from 35 years of publications in distance Education. Distance Education , 37 (3), 245–269.

Download references

Acknowledgements

No external funding/ NA.

Author information

Authors and affiliations.

Development for the JHB College of Business and Public Administration, 5500 University Parkway, San Bernardino, California, 92407, USA

Montgomery Van Wart, Anna Ni, Pamela Medina, Jesus Canelon, Melika Kordrostami, Jing Zhang & Yu Liu

You can also search for this author in PubMed   Google Scholar

Contributions

Equal. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Montgomery Van Wart .

Ethics declarations

Competing interests.

We have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Van Wart, M., Ni, A., Medina, P. et al. Integrating students’ perspectives about online learning: a hierarchy of factors. Int J Educ Technol High Educ 17 , 53 (2020). https://doi.org/10.1186/s41239-020-00229-8

Download citation

Received : 29 April 2020

Accepted : 30 July 2020

Published : 02 December 2020

DOI : https://doi.org/10.1186/s41239-020-00229-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Online education
• Online teaching
• Student perceptions
• Online quality
• Student presence

SYSTEMATIC REVIEW article

A systematic review of the effectiveness of online learning in higher education during the covid-19 pandemic period.

• 1 Department of Basic Education, Beihai Campus, Guilin University of Electronic Technology Beihai, Beihai, Guangxi, China
• 2 School of Sports and Arts, Harbin Sport University, Harbin, Heilongjiang, China
• 3 School of Music, Harbin Normal University, Harbin, Heilongjiang, China
• 4 School of General Education, Beihai Vocational College, Beihai, Guangxi, China
• 5 School of Economics and Management, Beihai Campus, Guilin University of Electronic Technology, Guilin, Guangxi, China

Background: The effectiveness of online learning in higher education during the COVID-19 pandemic period is a debated topic but a systematic review on this topic is absent.

Methods: The present study implemented a systematic review of 25 selected articles to comprehensively evaluate online learning effectiveness during the pandemic period and identify factors that influence such effectiveness.

Results: It was concluded that past studies failed to achieve a consensus over online learning effectiveness and research results are largely by how learning effectiveness was assessed, e.g., self-reported online learning effectiveness, longitudinal comparison, and RCT. Meanwhile, a set of factors that positively or negatively influence the effectiveness of online learning were identified, including infrastructure factors, instructional factors, the lack of social interaction, negative emotions, flexibility, and convenience.

Discussion: Although it is debated over the effectiveness of online learning during the pandemic period, it is generally believed that the pandemic brings a lot of challenges and difficulties to higher education and these challenges and difficulties are more prominent in developing countries. In addition, this review critically assesses limitations in past research, develops pedagogical implications, and proposes recommendations for future research.

1 Introduction

1.1 research background.

The COVID-19 pandemic first out broken in early 2020 has considerably shaped the higher education landscape globally. To restrain viral transmission, universities globally locked down, and teaching and learning activities were transferred to online platforms. Although online learning is a relatively mature learning model and is increasingly integrated into higher education, the sudden and unprepared transition to wholly online learning caused by the pandemic posed formidable challenges to higher education stakeholders, e.g., policymakers, instructors, and students, especially at the early stage of the pandemic ( García-Morales et al., 2021 ; Grafton-Clarke et al., 2022 ). Correspondingly, the effectiveness of online learning during the pandemic period is still questionable as online learning during this period has some unique characteristics, e.g., the lack of preparation, sudden and unprepared transition, the huge scale of implementation, and social distancing policies ( Sharma et al., 2020 ; Rahman, 2021 ; Tsang et al., 2021 ; Hollister et al., 2022 ; Zhang and Chen, 2023 ). This question is more prominent in developing or undeveloped countries because of insufficient Internet access, network problems, the lack of electronic devices, and poor network infrastructure ( Adnan and Anwar, 2020 ; Muthuprasad et al., 2021 ; Rahman, 2021 ; Chandrasiri and Weerakoon, 2022 ).

Learning effectiveness is a key consideration of education as it reflects the extent to which learning and teaching objectives are achieved and learners’ needs are satisfied ( Joy and Garcia, 2000 ; Swan, 2003 ). Online learning was generally proven to be effective within a higher education context ( Kebritchi et al., 2017 ) prior to the pandemic. ICTs have fundamentally shaped the process of learning as they allow learners to learn anywhere and anytime, interact with others efficiently and conveniently, and freely acquire a large volume of learning materials online ( Kebritchi et al., 2017 ; Choudhury and Pattnaik, 2020 ). Such benefits may be offset by the challenges brought about by the pandemic. A lot of empirical studies globally have investigated the effectiveness of online learning but there is currently a scarcity of a systematic review of these studies to comprehensively evaluate online learning effectiveness and identify factors that influence effectiveness.

At present, although the vast majority of countries have implemented opening policies to deal with the pandemic and higher education institutes have recovered offline teaching and learning, assessing the effectiveness of online learning during the pandemic period via a systematic review is still essential. First, it is necessary to summarize, learn from, and reflect on the lessons and experiences of online learning practices during the pandemic period to offer implications for future practices and research. Second, the review of online learning research carried out during the pandemic period is likely to generate interesting knowledge because of the unique research context. Third, higher education institutes still need a contingency plan for emergency online learning to deal with potential crises in the future, e.g., wars, pandemics, and natural disasters. A systematic review of research on the effectiveness of online learning during the pandemic period offers valuable knowledge for designing a contingency plan for the future.

1.2 Related concepts

1.2.1 online learning.

Online learning should not be simply understood as learning on the Internet or the integration of ICTs with learning because it is a systematic framework consisting of a set of pedagogies, technologies, implementations, and processes ( Kebritchi et al., 2017 ; Choudhury and Pattnaik, 2020). Choudhury and Pattnaik (2020; p.2) summarized prior definitions of online learning and provided a comprehensive and up-to-date definition, i.e., online learning refers to “ the transfer of knowledge and skills, in a well-designed course content that has established accreditations, through an electronic media like the Internet, Web 4.0, intranets and extranets .” Online learning differs from traditional learning because of not only technological differences, but also differences in social development and pedagogies ( Camargo et al., 2020 ). Online learning has also considerably shaped the patterns by which knowledge is stored, shared, and transferred, skills are practiced, as well as the way by which stakeholders (e.g., teachers and teachers) interact ( Desai et al., 2008 ; Anderson and Hajhashemi, 2013 ). In addition, online learning has altered educational objectives and learning requirements. Memorizing knowledge was traditionally viewed as vital to learning but it is now less important since required knowledge can be conveniently searched and acquired on the Internet while the reflection and application of knowledge becomes more important ( Gamage et al., 2023 ). Online learning also entails learners’ self-regulated learning ability more than traditional learning because the online learning environment inflicts less external regulation and provides more autonomy and flexibility ( Barnard-Brak et al., 2010 ; Wong et al., 2019 ). The above differences imply that traditional pedagogies may not apply to online learning.

There are a variety of online learning models according to the differences in learning methods, processes, outcomes, and the application of technologies ( Zeitoun, 2008 ). As ICTs can be used as either the foundation of learning or auxiliary means, online learning can be classified into assistant, blended, and wholly online models. Here, assistant online learning refers to the scenario where online learning technologies are used to supplement and support traditional learning; blended online learning refers to the integration/ mixture of online and offline methods, and; wholly online learning refers to the exclusive use of the Internet for learning ( Arkorful and Abaidoo, 2015 ). The present review focuses on wholly online learning because the review is interested in the COVID-19 pandemic context where learning activities are fully switched to online platforms.

1.2.2 Learning effectiveness

Learning effectiveness can be broadly defined as the extent to which learning and teaching objectives have been effectively and efficiently achieved via educational activities ( Swan, 2003 ) or the extent to which learners’ needs are satisfied by learning activities ( Joy and Garcia, 2000 ). It is a multi-dimensional construct because learning objectives and needs are always diversified ( Joy and Garcia, 2000 ; Swan, 2003 ). Assessing learning effectiveness is a key challenge in educational research and researchers generally use a set of subjective and objective indicators to assess learning effectiveness, e.g., examination scores, assignment performance, perceived effectiveness, student satisfaction, learning motivation, engagement in learning, and learning experience ( Rajaram and Collins, 2013 ; Noesgaard and Ørngreen, 2015 ). Prior research related to the effectiveness of online learning was diversified in terms of learning outcomes, e.g., satisfaction, perceived effectiveness, motivation, and learning engagement, and there is no consensus over which outcomes are valid indicators of learning effectiveness. The present study adopts a broad definition of learning effectiveness and considers various learning outcomes that are closely associated with learning objectives and needs.

1.3 Previous review research

Up to now, online learning during the COVID-19 pandemic period has attracted considerable attention from academia and there is a lot of related review research. Some review research analyzed the trends and major topics in related research. Pratama et al. (2020) tracked the trend of using online meeting applications in online learning during the pandemic period based on a systematic review of 12 articles. It was reported that the use of these applications kept a rising trend and this use helps promote learning and teaching processes. However, this review was descriptive and failed to identify problems related to these applications as well as the limitations of these applications. Zhang et al. (2022) implemented a bibliometric review to provide a holistic view of research on online learning in higher education during the COVID-19 pandemic period. They concluded that the majority of research focused on identifying the use of strategies and technologies, psychological impacts brought by the pandemic, and student perceptions. Meanwhile, collaborative learning, hands-on learning, discovery learning, and inquiry-based learning were the most frequently discussed instructional approaches. In addition, chemical and medical education were found to be the most investigated disciplines. This review hence offered a relatively comprehensive landscape of related research in the field. However, since it was a bibliometric review, it merely analyzed the superficial characteristics of past articles in the field without a detailed analysis of their research contributions. Bughrara et al. (2023) categorized the major research topics in the field of online medical education during the pandemic period via a scoping review. A total of 174 articles were included in the review and it was found there were seven major topics, including students’ mental health, stigma, student vaccination, use of telehealth, students’ physical health, online modifications and educational adaptations, and students’ attitudes and knowledge. Overall, the review comprehensively reveals major topics in the focused field.

Some scholars believed that online learning during the pandemic period has brought about a lot of problems while both students and teachers encounter many challenges. García-Morales et al. (2021) implemented a systematic review to identify the challenges encountered by higher education in an online learning scenario during the pandemic period. A total of seven studies were included and it was found that higher education suddenly transferred to online learning and a lot of technologies and platforms were used to support online learning. However, this transition was hasty and forced by the extreme situation. Thus, various stakeholders in learning and teaching (e.g., students, universities, and teachers) encountered difficulties in adapting to this sudden change. To deal with these challenges, universities need to utilize the potential of technologies, improve learning experience, and meet students’ expectations. The major limitation of García-Morales et al. (2021) review of the small-sized sample. Meanwhile, García-Morales et al. (2021) also failed to systematically categorize various types of challenges. Stojan et al. (2022) investigated the changes to medical education brought about by the shift to online learning in the COVID-19 pandemic context as well as the lessons and impacts of these changes via a systematic review. A total of 56 articles were included in the analysis, it was reported that small groups and didactics were the most prevalent instructional methods. Although learning engagement was always interactive, teachers majorly integrated technologies to amplify and replace, rather than transform learning. Based on this, they argued that the use of asynchronous and synchronous formats promoted online learning engagement and offered self-directed and flexible learning. The major limitation of this review is that the article is somewhat descriptive and lacks the crucial evaluation of problems of online learning.

Review research has also focused on the changes and impacts brought by online learning during the pandemic period. Camargo et al. (2020) implemented a meta-analysis on seven empirical studies regarding online learning methods during the pandemic period to evaluate feasible online learning platforms, effective online learning models, and the optimal duration of online lectures, as well as the perceptions of teachers and students in the online learning process. Overall, it was concluded that the shift from offline to online learning is feasible, and; effective online learning needs a well-trained and integrated team to identify students’ and teachers’ needs, timely respond, and support them via digital tools. In addition, the pandemic has brought more or less difficulties to online learning. An obvious limitation of this review is the overly small-sized sample ( N  = 7), which offers very limited information, but the review tries to answer too many questions (four questions). Grafton-Clarke et al. (2022) investigated the innovation/adaptations implemented, their impacts, and the reasons for their selections in the shift to online learning in medical education during the pandemic period via a systematic review of 55 articles. The major adaptations implemented include the rapid shift to the virtual space, pre-recorded videos or live streaming of surgical procedures, remote adaptations for clinical visits, and multidisciplinary ward rounds and team meetings. Major challenges encountered by students and teachers include the need for technical resources, faculty time, and devices, the shortage of standardized telemedicine curricula, and the lack of personal interactions. Based on this, they criticized the quality of online medical education. Tang (2023) explored the impact of the pandemic on primary, secondary, and tertiary education in the pandemic context via a systematic review of 41 articles. It was reported that the majority of these impacts are negative, e.g., learning loss among learners, assessment and experiential learning in the virtual environment, limitations in instructions, technology-related constraints, the lack of learning materials and resources, and deteriorated psychosocial well-being. These negative impacts are amplified by the unequal distribution of resources, unfair socioeconomic status, ethnicity, gender, physical conditions, and learning ability. Overall, this review comprehensively criticizes the problems brought about by online learning during the pandemic period.

Very little review research evaluated students’ responses to online learning during the pandemic period. For instance, Salas-Pilco et al. (2022) evaluated the engagement in online learning in Latin American higher education during the COVID-19 pandemic period via a systematic review of 23 studies. They considered three dimensions of engagement, including affective, cognitive, and behavioral engagement. They described the characteristics of learning engagement and proposed suggestions for enhancing engagement, including improving Internet connectivity, providing professional training, transforming higher education, ensuring quality, and offering emotional support. A key limitation of the review is that these authors focused on describing the characteristics of engagement without identifying factors that influence engagement.

A synthesis of previous review research offers some implications. First, although learning effectiveness is an important consideration in educational research, review research is scarce on this topic and hence there is a lack of comprehensive knowledge regarding the extent to which online learning is effective during the COVID-19 pandemic period. Second, according to past review research that summarized the major topics of related research, e.g., Bughrara et al. (2023) and Zhang et al. (2022) , the effectiveness of online learning is not a major topic in prior empirical research and hence the author of this article argues that this topic has not received due attention from researchers. Third, some review research has identified a lot of problems in online learning during the pandemic period, e.g., García-Morales et al. (2021) and Stojan et al. (2022) . Many of these problems are caused by the sudden and rapid shift to online learning as well as the unique context of the pandemic. These problems may undermine the effectiveness of online learning. However, the extent to which these problems influence online learning effectiveness is still under-investigated.

1.4 Purpose of the review research

The research is carried out based on a systematic review of past empirical research to answer the following two research questions:

Q1: To what extent online learning in higher education is effective during the COVID-19 pandemic period?

Q2: What factors shape the effectiveness of online learning in higher education during the COVID-19 pandemic period?

2 Research methodology

2.1 literature review as a research methodology.

Regardless of discipline, all academic research activities should be related to and based on existing knowledge. As a result, scholars must identify related research on the topic of interest, critically assess the quality and content of existing research, and synthesize available results ( Linnenluecke et al., 2020 ). However, this task is increasingly challenging for scholars because of the exponential growth of academic knowledge, which makes it difficult to be at the forefront and keep up with state-of-the-art research ( Snyder, 2019 ). Correspondingly, literature review, as a research methodology is more relevant than previously ( Snyder, 2019 ; Linnenluecke et al., 2020 ). A well-implemented review provides a solid foundation for facilitating theory development and advancing knowledge ( Webster and Watson, 2002 ). Here, a literature review is broadly defined as a more or less systematic way of collecting and synthesizing past studies ( Tranfield et al., 2003 ). It allows researchers to integrate perspectives and results from a lot of past research and is able to address research questions unanswered by a single study ( Snyder, 2019 ).

There are generally three types of literature review, including meta-analysis, bibliometric review, and systematic review ( Snyder, 2019 ). A meta-analysis refers to a statistical technique for integrating results from a large volume of empirical research (majorly quantitative research) to compare, identify, and evaluate patterns, relationships, agreements, and disagreements generated by research on the same topic ( Davis et al., 2014 ). This study does not adopt a meta-analysis for two reasons. First, the research on the effectiveness of online learning in the context of the COVID-19 pandemic was published since 2020 and currently, there is a limited volume of empirical evidence. If the study adopts a meta-analysis, the sample size will be small, resulting in limited statistical power. Second, as mentioned above, there are a variety of indicators, e.g., motivation, satisfaction, experience, test score, and perceived effectiveness ( Rajaram and Collins, 2013 ; Noesgaard and Ørngreen, 2015 ), that reflect different aspects of online learning effectiveness. The use of diversified effectiveness indicators increases the difficulty of carrying out meta-analysis.

A bibliometric review refers to the analysis of a large volume of empirical research in terms of publication characteristics (e.g., year, journal, and citation), theories, methods, research questions, countries, and authors ( Donthu et al., 2021 ) and it is useful in tracing the trend, distribution, relationship, and general patterns of research published in a focused topic ( Wallin, 2005 ). A bibliometric review does not fit the present study for two reasons. First, at present, there are less than 4 years of history of research on online learning effectiveness. Hence the volume of relevant research is limited and the public trend is currently unclear. Second, this study is interested in the inner content and results of articles published, rather than their external characteristics.

A systematic review is a method and process of critically identifying and appraising research in a specific field based on predefined inclusion and exclusion criteria to test a hypothesis, answer a research question, evaluate problems in past research, identify research gaps, and/or point out the avenue for future research ( Liberati et al., 2009 ; Moher et al., 2009 ). This type of review is particularly suitable to the present study as there are still a lot of unanswered questions regarding the effectiveness of online learning in the pandemic context, a need for indicating future research direction, a lack of summary of relevant research in this field, and a scarcity of critical appraisal of problems in past research.

Adopting a systematic review methodology brings multiple benefits to the present study. First, it is helpful for distinguishing what needs to be done from what has been done, identifying major contributions made by past research, finding out gaps in past research, avoiding fruitless research, and providing insights for future research in the focused field ( Linnenluecke et al., 2020 ). Second, it is also beneficial for finding out new research directions, needs for theory development, and potential solutions for limitations in past research ( Snyder, 2019 ). Third, this methodology helps scholars to efficiently gain an overview of valuable research results and theories generated by past research, which inspires their research design, ideas, and perspectives ( Callahan, 2014 ).

Commonly, a systematic review can be either author-centric or theme-centric ( Webster and Watson, 2002 ) and the present review is theme-centric. Specifically, an author-centric review focuses on works published by a certain author or a group of authors and summarizes the major contributions made by the author(s; ( Webster and Watson, 2002 ). This type of review is problematic in terms of its incompleteness of research conclusions in a specific field and descriptive nature ( Linnenluecke et al., 2020 ). A theme-centric review is more common where a researcher guides readers through reviewing themes, concepts, and interesting phenomena according to a certain logic ( Callahan, 2014 ). A theme in this review can be further structured into several related sub-themes and this type of review helps researchers to gain a comprehensive understanding of relevant academic knowledge ( Papaioannou et al., 2016 ).

2.2 Research procedures

This study follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guideline ( Liberati et al., 2009 ) to implement a systematic review. The guideline indicates four phases of performing a systematic review, including (1) identifying possible research, (2) abstract screening, (3) assessing full-text for eligibility, and (4) qualitatively synthesizing included research. Figure 1 provides a flowchart of the process and the number of articles excluded and included in each phase.

Figure 1 . PRISMA flowchart concerning the selection of articles.

This study uses multiple academic databases to identify possible research, e.g., Academic Search Complete, IGI Global, ACM Digital Library, Elsevier (SCOPUS), Emerald, IEEE Xplore, Web of Science, Science Direct, ProQuest, Wiley Online Library, Taylor and Francis, and EBSCO. Since the COVID-19 pandemic broke out in January 2020, this study limits the literature search to articles published from January 2020 to August 2023. During this period, online learning was highly prevalent in schools globally and a considerable volume of articles were published to investigate various aspects of online learning in this period. Keywords used for searching possible research include pandemic, COVID, SARS-CoV-2, 2019-nCoV, coronavirus, online learning, e-learning, electronic learning, higher education, tertiary education, universities, learning effectiveness, learning satisfaction, learning engagement, and learning motivation. Aside from searching from databases, this study also manually checks the reference lists of relevant articles and uses Google Scholar to find out other articles that have cited these articles.

2.3 Inclusion and exclusion criteria

Articles included in the review must meet the following criteria. First, articles have to be written in English and published on peer-reviewed journals. The academic language being English was chosen because it is in the Q zone of the specified search engines. Second, the research must be carried out in an online learning context. Third, the research must have collected and analyzed empirical data. Fourth, the research should be implemented in a higher education context and during the pandemic period. Fifth, the outcome variable must be factors related to learning effectiveness, and included studies must have reported the quantitative results for online learning effectiveness. The outcome variable should be measured by data collected from students, rather than other individuals (e.g., instructors). For instance, the research of Rahayu and Wirza (2020) used teacher perception as a measurement of online learning effectiveness and was hence excluded from the sample. According to the above criteria, a total of 25 articles were included in the review.

2.4 Data extraction and analysis

Content analysis is performed on included articles and an inductive approach is used to answer the two research questions. First, to understand the basic characteristics of the 25 articles/studies, the researcher summarizes their types, research designs, and samples and categorizes them into several groups. The researcher carefully reads the full-text of these articles and codes valuable pieces of content. In this process, an inductive approach is used, and key themes in these studies have been extracted and summarized. Second, the researcher further categorizes these studies into different groups according to their similarities and differences in research findings. In this way, these studies are broadly categorized into three groups, i.e., (1) ineffective (2) neutral, and (3) effective. Based on this, the research answers the research question and indicates the percentage of studies that evidenced online learning as effective in a COVID-19 pandemic context. The researcher also discusses how online learning is effective by analyzing the learning outcomes brought by online learning. Third, the researcher analyzes and compares the characteristics of the three groups of studies and extracts key themes that are relevant to the conditional effectiveness of online learning from these studies. Based on this, the researcher identifies factors that influence the effectiveness of online learning in a pandemic context. In this way, the two research questions have been adequately answered.

3 Research results and discussion

3.1 study characteristics.

Table 1 shows the statistics of the 25 studies while Table 2 shows a summary of these studies. Overall, these studies varied greatly in terms of research design, research subjects, contexts, measurements of learning effectiveness, and eventually research findings. Approximately half of the studies were published in 2021 and the number of studies reduced in 2022 and 2023, which may be attributed to the fact that universities gradually implemented opening-up policies after 2020. China received the largest number of studies ( N  = 5), followed by India ( N = 4) and the United States ( N  = 3). The sample sizes of the majority of studies (88.0%) ranged between 101 and 500. As this review excluded qualitative studies, all studies included adopted a purely quantitative design (88.0%) or a mixed design (12.0%). The majority of the studies were cross-sectional (72%) and a few studies (2%) were experimental.

Table 1 . Statistics of studies included in the review.

Table 2 . A summary of studies reviewed.

3.2 The effectiveness of online learning

Overall, the 25 studies generated mixed results regarding the effectiveness of online learning during the pandemic period. 9 (36%) studies reported online learning as effective; 13 (52%) studies reported online learning as ineffective, and the rest 3 (12%) studies produced neutral results. However, it should be noted that the results generated by these studies are not comparable as they used different approaches to evaluate the effectiveness of online learning. According to the approach of evaluating online learning effectiveness, these studies are categorized into four groups, including (1) Cross-sectional evaluation of online learning effectiveness without a comparison with offline learning; without a control group ( N  = 14; 56%), (2) Cross-sectional comparison of the effectiveness of online learning with offline learning; without control group (7; 28%), (3) Longitudinal comparison of the effectiveness of online learning with offline learning, without a control group ( N  = 2; 8%), and (4) Randomized Controlled Trial (RCT); with a control group ( N  = 2; 8%).

The first group of studies asked students to report the extent to which they perceived online learning as effective, they had achieved expected learning outcomes through online learning, or they were satisfied with online learning experience or outcomes, without a comparison with offline learning. Six out of 14 studies reported online learning as ineffective, including Adnan and Anwar (2020) , Hong et al. (2021) , Mok et al. (2021) , Baber (2022) , Chandrasiri and Weerakoon (2022) , and Lalduhawma et al. (2022) . Five out of 14 studies reported online learning as effective, including Almusharraf and Khahro (2020) , Sharma et al. (2020) , Mahyoob (2021) , Rahman (2021) , and Haningsih and Rohmi (2022) . In addition, 3 out of 14 studies reported neutral results, including Cranfield et al. (2021) , Tsang et al. (2021) , and Conrad et al. (2022) . It should be noted that this measurement approach is problematic in three aspects. First, researchers used various survey instruments to measure learning effectiveness without reaching a consensus over a widely accepted instrument. As a result, these studies measured different aspects of learning effectiveness and hence their results may be incomparable. Second, these studies relied on students’ self-reports to evaluate learning effectiveness, which is subjective and inaccurate. Third, even though students perceived online learning as effective, it does not imply that online learning is more effective than offline learning because of the absence of comparables.

The second group of studies asked students to compare online learning with offline learning to evaluate learning effectiveness. Interestingly, all 7 studies, including Alawamleh et al. (2020) , Almahasees et al. (2021) , Gonzalez-Ramirez et al. (2021) , Muthuprasad et al. (2021) , Selco and Habbak (2021) , Hollister et al. (2022) , and Zhang and Chen (2023) , reported that online learning was perceived by participants as less effective than offline learning. It should be noted that these results were specific to the COVID-19 pandemic context where strict social distancing policies were implemented. Consequently, these results should be interpreted as online learning during the school lockdown period was perceived by participants as less effective than offline learning during the pre-pandemic period. A key problem of the measurement of learning effectiveness in these studies is subjectivity, i.e., students’ self-reported online learning effectiveness relative to offline learning may be subjective and influenced by a lot of factors caused by the pandemic, e.g., negative emotions (e.g., fear, loneliness, and anxiety).

Only two studies implemented a longitudinal comparison of the effectiveness of online learning with offline learning, i.e., Chang et al. (2021) and Fyllos et al. (2021) . Interestingly, both studies reported that participants perceived online learning as more effective than offline learning, which is contradicted with the second group of studies. In the two studies, the same group of students participated in offline learning and online learning successively and rated the effectiveness of the two learning approaches, respectively. The two studies were implemented by time coincidence, i.e., researchers unexpectedly encountered the pandemic and subsequently, school lockdown when they were investigating learning effectiveness. Such time coincidence enabled them to compare the effectiveness of offline and online learning. However, this research design has three key problems. First, the content of learning in the online and offline learning periods was different and hence the evaluations of learning effectiveness of the two periods are not comparable. Second, self-reported learning effectiveness is subjective. Third, students are likely to obtain better examination scores in online examinations than in offline examinations because online examinations bring a lot of cheating behaviors and are less fair than offline examinations. As reported by Fyllos et al. (2021) , the examination score after online learning was significantly higher than after offline learning. Chang et al. (2021) reported that participants generally believed that offline examinations are fairer than online examinations.

Lastly, only two studies, i.e., Jiang et al. (2023) and Shirahmadi et al. (2023) , implemented an RCT design, which is more persuasive, objective, and accurate than the above-reviewed studies. Indeed, implementing an RCT to evaluate the effectiveness of online learning was a formidable challenge during the pandemic period because of viral transmission and social distancing policies. Both studies reported that online learning is more effective than offline learning during the pandemic period. However, it is questionable about the extent to which such results are affected by health/safety-related issues. It is reasonable to infer that online learning was perceived by students as safer than offline learning during the pandemic period and such perceptions may affect learning effectiveness.

Overall, it is difficult to conclude whether online learning is effective during the pandemic period. Nevertheless, it is possible to identify factors that shape the effectiveness of online learning, which is discussed in the next section.

3.3 Factors that shape online learning effectiveness

Infrastructure factors were reported as the most salient factors that determine online learning effectiveness. It seems that research from developed countries generated more positive results for online learning than research from less developed countries. This view was confirmed by the cross-country comparative study of Cranfield et al. (2021) . Indeed, online learning entails the support of ICT infrastructure, and hence ICT related factors, e.g., Internet connectivity, technical issues, network speed, accessibility of digital devices, and digital devices, considerably influence the effectiveness of online learning ( García-Morales et al., 2021 ; Grafton-Clarke et al., 2022 ). Prior review research, e.g., Tang (2023) also suggested that the unequal distribution of resources and unfair socioeconomic status intensified the problems brought about by online learning during the pandemic period. Salas-Pilco et al. (2022) recommended that improving Internet connectivity would increase students’ engagement in online learning during the pandemic period.

Adnan and Anwar (2020) study is one of the most cited works in the focused field. They reported that online learning is ineffective in Pakistan because of the problems of Internet access due to monetary and technical issues. The above problems hinder students from implementing online learning activities, making online learning ineffective. Likewise, Lalduhawma et al. (2022) research from India indicated that online learning is ineffective because of poor network interactivity, slow data speed, low data limits, and expensive costs of devices. As a result, online learning during the COVID-19 pandemic may have expanded the education gap between developed and developing countries because of developing countries’ infrastructure disadvantages. More attention to online learning infrastructure problems in developing countries is needed.

Instructional factors, e.g., course management and design, instructor characteristics, instructor-student interaction, assignments, and assessments were found to affect online learning effectiveness ( Sharma et al., 2020 ; Rahman, 2021 ; Tsang et al., 2021 ; Hollister et al., 2022 ; Zhang and Chen, 2023 ). Although these instructional factors have been well-documented as significant drivers of learning effectiveness in traditional learning literature, these factors in the pandemic period have some unique characteristics. Both students and teachers were not well prepared for wholly online instruction and learning in 2020 and hence they encountered a lot of problems in course management and design, learning interactions, assignments, and assessments ( Stojan et al., 2022 ; Tang, 2023 ). García-Morales et al. (2021) review also suggested that various stakeholders in learning and teaching encountered difficulties in adapting to the sudden, hasty, and forced transition of offline to online learning. Consequently, these instructional factors become salient in terms of affecting online learning effectiveness.

The negative role of the lack of social interaction caused by social distancing in affecting online learning effectiveness was highlighted by a lot of studies ( Almahasees et al., 2021 ; Baber, 2022 ; Conrad et al., 2022 ; Hollister et al., 2022 ). Baber (2022) argued that people give more importance to saving lives than socializing in the online environment and hence social interactions in learning are considerably reduced by social distancing norms. The negative impact of the lack of social interaction on online learning effectiveness is reflected in two aspects. First, according to a constructivist view, interaction is an indispensable element of learning because knowledge is actively constructed by learners in social interactions ( Woo and Reeves, 2007 ). Consequently, online learning effectiveness during the pandemic period is reduced by the lack of social interaction. Second, the lack of social interaction brings a lot of negative emotions, e.g., feelings of isolation, loneliness, anxiety, and depression ( Alawamleh et al., 2020 ; Gonzalez-Ramirez et al., 2021 ; Selco and Habbak, 2021 ). Such negative emotions undermine online learning effectiveness.

Negative emotions caused by the pandemic and school lockdown were also found to be detrimental to online learning effectiveness. In this context, it was reported that many students experience a lot of negative emotions, e.g., feelings of isolation, exhaustion, loneliness, and distraction ( Alawamleh et al., 2020 ; Gonzalez-Ramirez et al., 2021 ; Selco and Habbak, 2021 ). Such negative emotions, as mentioned above, reduce online learning effectiveness.

Several factors were also found to increase online learning effectiveness during the pandemic period, e.g., convenience and flexibility ( Hong et al., 2021 ; Muthuprasad et al., 2021 ; Selco and Habbak, 2021 ). Students with strong self-regulated learning abilities gain more benefits from convenience and flexibility in online learning ( Hong et al., 2021 ).

Overall, although it is debated over the effectiveness of online learning during the pandemic period, it is generally believed that the pandemic brings a lot of challenges and difficulties to higher education. Meanwhile, the majority of students prefer offline learning to online learning. The above challenges and difficulties are more prominent in developing countries than in developed countries.

3.4 Pedagogical implications

The results generated by the systematic review offer a lot of pedagogical implications. First, online learning entails the support of ICT infrastructure, and infrastructure defects strongly undermine learning effectiveness ( García-Morales et al., 2021 ; Grafton-Clarke et al., 2022 ). Given the fact online learning is increasingly integrated into higher education ( Kebritchi et al., 2017 ) regardless of the presence of the pandemic, governments globally should increase the investment in learning-related ICT infrastructure in higher education institutes. Meanwhile, schools should consider students’ affordability of digital devices and network fees when implementing online learning activities. It is important to offer material support for those students with poor economic status. Infrastructure issues are more prominent in developing countries because of the lack of monetary resources and poor infrastructure base. Thus, international collaboration and aid are recommended to address these issues.

Second, since the lack of social interaction is a key factor that reduces online learning effectiveness, it is important to increase social interactions during the implementation of online learning activities. On the one hand, both students and instructors are encouraged to utilize network technologies to promote inter-individual interactions. On the other hand, the two parties are also encouraged to engage in offline interaction activities if the risk is acceptable.

Third, special attention should be paid to students’ emotions during the online learning process as online learning may bring a lot of negative emotions to students, which undermine learning effectiveness ( Alawamleh et al., 2020 ; Gonzalez-Ramirez et al., 2021 ; Selco and Habbak, 2021 ). In addition, higher education institutes should prepare a contingency plan for emergency online learning to deal with potential crises in the future, e.g., wars, pandemics, and natural disasters.

3.5 Limitations and suggestions for future research

There are several limitations in past research regarding online learning effectiveness during the pandemic period. The first is the lack of rigor in assessing learning effectiveness. Evidently, there is a scarcity of empirical research with an RCT design, which is considered to be accurate, objective, and rigorous in assessing pedagogical models ( Torgerson and Torgerson, 2001 ). The scarcity of ICT research leads to the difficulty in accurately assessing the effectiveness of online learning and comparing it with offline learning. Second, the widely accepted criteria for assessing learning effectiveness are absent, and past empirical studies used diversified procedures, techniques, instruments, and criteria for measuring online learning effectiveness, resulting in difficulty in comparing research results. Third, learning effectiveness is a multi-dimensional construct but its multidimensionality was largely ignored by past research. Therefore, it is difficult to evaluate which dimensions of learning effectiveness are promoted or undermined by online learning and it is also difficult to compare the results of different studies. Finally, there is very limited knowledge about the difference in online learning effectiveness between different subjects. It is likely that the subjects that depend on lab-based work (e.g., experimental physics, organic chemistry, and cell biology) are less appropriate for online learning than the subjects that depend on desk-based work (e.g., economics, psychology, and literature).

To deal with the above limitations, there are several recommendations for future research on online learning effectiveness. First, future research is encouraged to adopt an RCT design and collect a large-sized sample to objectively, rigorously, and accurately quantify the effectiveness of online learning. Second, scholars are also encouraged to develop a new framework to assess learning effectiveness comprehensively. This framework should cover multiple dimensions of learning effectiveness and have strong generalizability. Finally, it is recommended that future research could compare the effectiveness of online learning between different subjects.

4 Conclusion

This study carried out a systematic review of 25 empirical studies published between 2020 and 2023 to evaluate the effectiveness of online learning during the COVID-19 pandemic period. According to how online learning effectiveness was assessed, these 25 studies were categorized into four groups. The first group of studies employed a cross-sectional design and assessed online learning based on students’ perceptions without a control group. Less than half of these studies reported online learning as effective. The second group of studies also employed a cross-sectional design and asked students to compare the effectiveness of online learning with offline learning. All these studies reported that online learning is less effective than offline learning. The third group of studies employed a longitudinal design and compared the effectiveness of online learning with offline learning but without a control group and this group includes only 2 studies. It was reported that online learning is more effective than offline learning. The fourth group of studies employed an RCT design and this group includes only 2 studies. Both studies reported online learning as more effective than offline learning.

Overall, it is difficult to conclude whether online learning is effective during the pandemic period because of the diversified research contexts, methods, and approaches in past research. Nevertheless, the review identifies a set of factors that positively or negatively influence the effectiveness of online learning, including infrastructure factors, instructional factors, the lack of social interaction, negative emotions, flexibility, and convenience. Although it is debated over the effectiveness of online learning during the pandemic period, it is generally believed that the pandemic brings a lot of challenges and difficulties to higher education. Meanwhile, the majority of students prefer offline learning to online learning. In addition, developing countries face more challenges and difficulties in online learning because of monetary and infrastructure issues.

The findings of this review offer significant pedagogical implications for online learning in higher education institutes, including enhancing the development of ICT infrastructure, providing material support for students with poor economic status, enhancing social interactions, paying attention to students’ emotional status, and preparing a contingency plan of emergency online learning.

The review also identifies several limitations in past research regarding online learning effectiveness during the pandemic period, including the lack of rigor in assessing learning effectiveness, the absence of accepted criteria for assessing learning effectiveness, the neglect of the multidimensionality of learning effectiveness, and limited knowledge about the difference in online learning effectiveness between different subjects.

To deal with the above limitations, there are several recommendations for future research on online learning effectiveness. First, future research is encouraged to adopt an RCT design and collect a large-sized sample to objectively, rigorously, and accurately quantify the effectiveness of online learning. Second, scholars are also encouraged to develop a new framework to assess learning effectiveness comprehensively. This framework should cover multiple dimensions of learning effectiveness and have strong generalizability. Finally, it is recommended that future research could compare the effectiveness of online learning between different subjects. To fix these limitations in future research, recommendations are made.

It should be noted that this review is not free of problems. First, only studies that quantitatively measured online learning effectiveness were included in the review and hence a lot of other studies (e.g., qualitative studies) that investigated factors that influence online learning effectiveness were excluded, resulting in a relatively small-sized sample and incomplete synthesis of past research contributions. Second, since this review was qualitative, it was difficult to accurately quantify the level of online learning effectiveness.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

WM: Writing – original draft, Writing – review & editing. LY: Writing – original draft, Writing – review & editing. CL: Writing – review & editing. NP: Writing – review & editing. XP: Writing – review & editing. YZ: Writing – review & editing.

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Adnan, M., and Anwar, K. (2020). Online learning amid the COVID-19 pandemic: Students' perspectives. J. Pedagogical Sociol. Psychol. 1, 45–51. doi: 10.33902/JPSP.2020261309

Crossref Full Text | Google Scholar

Alawamleh, M., Al-Twait, L. M., and Al-Saht, G. R. (2020). The effect of online learning on communication between instructors and students during Covid-19 pandemic. Asian Educ. Develop. Stud. 11, 380–400. doi: 10.1108/AEDS-06-2020-0131

Almahasees, Z., Mohsen, K., and Amin, M. O. (2021). Faculty’s and students’ perceptions of online learning during COVID-19. Front. Educ. 6:638470. doi: 10.3389/feduc.2021.638470

Almusharraf, N., and Khahro, S. (2020). Students satisfaction with online learning experiences during the COVID-19 pandemic. Int. J. Emerg. Technol. Learn. (iJET) 15, 246–267. doi: 10.3991/ijet.v15i21.15647

Anderson, N., and Hajhashemi, K. (2013). Online learning: from a specialized distance education paradigm to a ubiquitous element of contemporary education. In 4th international conference on e-learning and e-teaching (ICELET 2013) (pp. 91–94). IEEE.

Google Scholar

Arkorful, V., and Abaidoo, N. (2015). The role of e-learning, advantages and disadvantages of its adoption in higher education. Int. J. Instructional Technol. Distance Learn. 12, 29–42.

Baber, H. (2022). Social interaction and effectiveness of the online learning–a moderating role of maintaining social distance during the pandemic COVID-19. Asian Educ. Develop. Stud. 11, 159–171. doi: 10.1108/AEDS-09-2020-0209

Barnard-Brak, L., Paton, V. O., and Lan, W. Y. (2010). Profiles in self-regulated learning in the online learning environment. Int. Rev. Res. Open Dist. Learn. 11, 61–80. doi: 10.19173/irrodl.v11i1.769

Bughrara, M. S., Swanberg, S. M., Lucia, V. C., Schmitz, K., Jung, D., and Wunderlich-Barillas, T. (2023). Beyond COVID-19: the impact of recent pandemics on medical students and their education: a scoping review. Med. Educ. Online 28:2139657. doi: 10.1080/10872981.2022.2139657

PubMed Abstract | Crossref Full Text | Google Scholar

Callahan, J. L. (2014). Writing literature reviews: a reprise and update. Hum. Resour. Dev. Rev. 13, 271–275. doi: 10.1177/1534484314536705

Camargo, C. P., Tempski, P. Z., Busnardo, F. F., Martins, M. D. A., and Gemperli, R. (2020). Online learning and COVID-19: a meta-synthesis analysis. Clinics 75:e2286. doi: 10.6061/clinics/2020/e2286

Choudhury, S., and Pattnaik, S. (2020). Emerging themes in e-learning: A review from the stakeholders’ perspective. Computers and Education 144, 103657. doi: 10.1016/j.compedu.2019.103657

Chandrasiri, N. R., and Weerakoon, B. S. (2022). Online learning during the COVID-19 pandemic: perceptions of allied health sciences undergraduates. Radiography 28, 545–549. doi: 10.1016/j.radi.2021.11.008

Chang, J. Y. F., Wang, L. H., Lin, T. C., Cheng, F. C., and Chiang, C. P. (2021). Comparison of learning effectiveness between physical classroom and online learning for dental education during the COVID-19 pandemic. J. Dental Sci. 16, 1281–1289. doi: 10.1016/j.jds.2021.07.016

Conrad, C., Deng, Q., Caron, I., Shkurska, O., Skerrett, P., and Sundararajan, B. (2022). How student perceptions about online learning difficulty influenced their satisfaction during Canada's Covid-19 response. Br. J. Educ. Technol. 53, 534–557. doi: 10.1111/bjet.13206

Cranfield, D. J., Tick, A., Venter, I. M., Blignaut, R. J., and Renaud, K. (2021). Higher education students’ perceptions of online learning during COVID-19—a comparative study. Educ. Sci. 11, 403–420. doi: 10.3390/educsci11080403

Desai, M. S., Hart, J., and Richards, T. C. (2008). E-learning: paradigm shift in education. Education 129, 1–20.

Davis, J., Mengersen, K., Bennett, S., and Mazerolle, L. (2014). Viewing systematic reviews and meta-analysis in social research through different lenses. SpringerPlus 3, 1–9. doi: 10.1186/2193-1801-3-511

Donthu, N., Kumar, S., Mukherjee, D., Pandey, N., and Lim, W. M. (2021). How to conduct a bibliometric analysis: An overview and guidelines. Journal of business research 133, 264–269. doi: 10.1016/j.jbusres.2021.04.070

Fyllos, A., Kanellopoulos, A., Kitixis, P., Cojocari, D. V., Markou, A., Raoulis, V., et al. (2021). University students perception of online education: is engagement enough? Acta Informatica Medica 29, 4–9. doi: 10.5455/aim.2021.29.4-9

Gamage, D., Ruipérez-Valiente, J. A., and Reich, J. (2023). A paradigm shift in designing education technology for online learning: opportunities and challenges. Front. Educ. 8:1194979. doi: 10.3389/feduc.2023.1194979

García-Morales, V. J., Garrido-Moreno, A., and Martín-Rojas, R. (2021). The transformation of higher education after the COVID disruption: emerging challenges in an online learning scenario. Front. Psychol. 12:616059. doi: 10.3389/fpsyg.2021.616059

Gonzalez-Ramirez, J., Mulqueen, K., Zealand, R., Silverstein, S., Mulqueen, C., and BuShell, S. (2021). Emergency online learning: college students' perceptions during the COVID-19 pandemic. Coll. Stud. J. 55, 29–46.

Grafton-Clarke, C., Uraiby, H., Gordon, M., Clarke, N., Rees, E., Park, S., et al. (2022). Pivot to online learning for adapting or continuing workplace-based clinical learning in medical education following the COVID-19 pandemic: a BEME systematic review: BEME guide no. 70. Med. Teach. 44, 227–243. doi: 10.1080/0142159X.2021.1992372

Haningsih, S., and Rohmi, P. (2022). The pattern of hybrid learning to maintain learning effectiveness at the higher education level post-COVID-19 pandemic. Eurasian J. Educ. Res. 11, 243–257. doi: 10.12973/eu-jer.11.1.243

Hollister, B., Nair, P., Hill-Lindsay, S., and Chukoskie, L. (2022). Engagement in online learning: student attitudes and behavior during COVID-19. Front. Educ. 7:851019. doi: 10.3389/feduc.2022.851019

Hong, J. C., Lee, Y. F., and Ye, J. H. (2021). Procrastination predicts online self-regulated learning and online learning ineffectiveness during the coronavirus lockdown. Personal. Individ. Differ. 174:110673. doi: 10.1016/j.paid.2021.110673

Jiang, P., Namaziandost, E., Azizi, Z., and Razmi, M. H. (2023). Exploring the effects of online learning on EFL learners’ motivation, anxiety, and attitudes during the COVID-19 pandemic: a focus on Iran. Curr. Psychol. 42, 2310–2324. doi: 10.1007/s12144-022-04013-x

Joy, E. H., and Garcia, F. E. (2000). Measuring learning effectiveness: a new look at no-significant-difference findings. JALN 4, 33–39.

Kebritchi, M., Lipschuetz, A., and Santiague, L. (2017). Issues and challenges for teaching successful online courses in higher education: a literature review. J. Educ. Technol. Syst. 46, 4–29. doi: 10.1177/0047239516661713

Lalduhawma, L. P., Thangmawia, L., and Hussain, J. (2022). Effectiveness of online learning during the COVID-19 pandemic in Mizoram. J. Educ. e-Learning Res. 9, 175–183. doi: 10.20448/jeelr.v9i3.4162

Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P., et al. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Annals of internal medicine , 151, W-65. doi: 10.7326/0003-4819-151-4-200908180-00136

Linnenluecke, M. K., Marrone, M., and Singh, A. K. (2020). Conducting systematic literature reviews and bibliometric analyses. Aust. J. Manag. 45, 175–194. doi: 10.1177/0312896219877678

Mahyoob, M. (2021). Online learning effectiveness during the COVID-19 pandemic: a case study of Saudi universities. Int. J. Info. Commun. Technol. Educ. (IJICTE) 17, 1–14. doi: 10.4018/IJICTE.20211001.oa7

Moher, D., Liberati, A., Tetzlaff, D., and Altman, G. and PRISMA Group (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of internal medicine , 151, 264–269. doi: 10.3736/jcim20090918

Mok, K. H., Xiong, W., and Bin Aedy Rahman, H. N. (2021). COVID-19 pandemic’s disruption on university teaching and learning and competence cultivation: student evaluation of online learning experiences in Hong Kong. Int. J. Chinese Educ. 10:221258682110070. doi: 10.1177/22125868211007011

Muthuprasad, T., Aiswarya, S., Aditya, K. S., and Jha, G. K. (2021). Students’ perception and preference for online education in India during COVID-19 pandemic. Soc. Sci. Humanities open 3:100101. doi: 10.1016/j.ssaho.2020.100101

Noesgaard, S. S., and Ørngreen, R. (2015). The effectiveness of e-learning: an explorative and integrative review of the definitions, methodologies and factors that promote e-learning effectiveness. Electronic J. E-learning 13, 278–290.

Papaioannou, D., Sutton, A., and Booth, A. (2016). Systematic approaches to a successful literature review. London: Sage.

Pratama, H., Azman, M. N. A., Kassymova, G. K., and Duisenbayeva, S. S. (2020). The trend in using online meeting applications for learning during the period of pandemic COVID-19: a literature review. J. Innovation in Educ. Cultural Res. 1, 58–68. doi: 10.46843/jiecr.v1i2.15

Rahayu, R. P., and Wirza, Y. (2020). Teachers’ perception of online learning during pandemic covid-19. Jurnal penelitian pendidikan 20, 392–406. doi: 10.17509/jpp.v20i3.29226

Rahman, A. (2021). Using students’ experience to derive effectiveness of COVID-19-lockdown-induced emergency online learning at undergraduate level: evidence from Assam. India. Higher Education for the Future 8, 71–89. doi: 10.1177/2347631120980549

Rajaram, K., and Collins, B. (2013). Qualitative identification of learning effectiveness indicators among mainland Chinese students in culturally dislocated study environments. J. Int. Educ. Bus. 6, 179–199. doi: 10.1108/JIEB-03-2013-0010

Salas-Pilco, S. Z., Yang, Y., and Zhang, Z. (2022). Student engagement in online learning in Latin American higher education during the COVID-19 pandemic: a systematic review. Br. J. Educ. Technol. 53, 593–619. doi: 10.1111/bjet.13190

Selco, J. I., and Habbak, M. (2021). Stem students’ perceptions on emergency online learning during the covid-19 pandemic: challenges and successes. Educ. Sci. 11:799. doi: 10.3390/educsci11120799

Sharma, K., Deo, G., Timalsina, S., Joshi, A., Shrestha, N., and Neupane, H. C. (2020). Online learning in the face of COVID-19 pandemic: assessment of students’ satisfaction at Chitwan medical college of Nepal. Kathmandu Univ. Med. J. 18, 40–47. doi: 10.3126/kumj.v18i2.32943

Shirahmadi, S., Hazavehei, S. M. M., Abbasi, H., Otogara, M., Etesamifard, T., Roshanaei, G., et al. (2023). Effectiveness of online practical education on vaccination training in the students of bachelor programs during the Covid-19 pandemic. PLoS One 18:e0280312. doi: 10.1371/journal.pone.0280312

Snyder, H. (2019). Literature review as a research methodology: an overview and guidelines. J. Bus. Res. 104, 333–339. doi: 10.1016/j.jbusres.2019.07.039

Stojan, J., Haas, M., Thammasitboon, S., Lander, L., Evans, S., Pawlik, C., et al. (2022). Online learning developments in undergraduate medical education in response to the COVID-19 pandemic: a BEME systematic review: BEME guide no. 69. Med. Teach. 44, 109–129. doi: 10.1080/0142159X.2021.1992373

Swan, K. (2003). Learning effectiveness online: what the research tells us. Elements of quality online education, practice and direction 4, 13–47.

Tang, K. H. D. (2023). Impacts of COVID-19 on primary, secondary and tertiary education: a comprehensive review and recommendations for educational practices. Educ. Res. Policy Prac. 22, 23–61. doi: 10.1007/s10671-022-09319-y

Torgerson, C. J., and Torgerson, D. J. (2001). The need for randomised controlled trials in educational research. Br. J. Educ. Stud. 49, 316–328. doi: 10.1111/1467-8527.t01-1-00178

Tranfield, D., Denyer, D., and Smart, P. (2003). Towards a methodology for developing evidence‐informed management knowledge by means of systematic review. British journal of management , 14, 207–222. doi: 10.1111/1467-8551.00375

Tsang, J. T., So, M. K., Chong, A. C., Lam, B. S., and Chu, A. M. (2021). Higher education during the pandemic: the predictive factors of learning effectiveness in COVID-19 online learning. Educ. Sci. 11:446. doi: 10.3390/educsci11080446

Wallin, J. A. (2005). Bibliometric methods: pitfalls and possibilities. Basic Clin. Pharmacol. Toxicol. 97, 261–275. doi: 10.1111/j.1742-7843.2005.pto_139.x

Webster, J., and Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS quarterly , 26, 13–23.

Wong, J., Baars, M., Davis, D., Van Der Zee, T., Houben, G. J., and Paas, F. (2019). Supporting self-regulated learning in online learning environments and MOOCs: a systematic review. Int. J. Human–Computer Interaction 35, 356–373. doi: 10.1080/10447318.2018.1543084

Woo, Y., and Reeves, T. C. (2007). Meaningful interaction in web-based learning: a social constructivist interpretation. Internet High. Educ. 10, 15–25. doi: 10.1016/j.iheduc.2006.10.005

Zeitoun, H. (2008). E-learning: Concept, Issues, Application, Evaluation . Riyadh: Dar Alsolateah Publication.

Zhang, L., Carter, R. A. Jr., Qian, X., Yang, S., Rujimora, J., and Wen, S. (2022). Academia's responses to crisis: a bibliometric analysis of literature on online learning in higher education during COVID-19. Br. J. Educ. Technol. 53, 620–646. doi: 10.1111/bjet.13191

Zhang, Y., and Chen, X. (2023). Students’ perceptions of online learning in the post-COVID era: a focused case from the universities of applied sciences in China. Sustain. For. 15:946. doi: 10.3390/su15020946

Keywords: COVID-19 pandemic, higher education, online learning, learning effectiveness, systematic review

Citation: Meng W, Yu L, Liu C, Pan N, Pang X and Zhu Y (2024) A systematic review of the effectiveness of online learning in higher education during the COVID-19 pandemic period. Front. Educ . 8:1334153. doi: 10.3389/feduc.2023.1334153

Received: 06 November 2023; Accepted: 27 December 2023; Published: 17 January 2024.

Reviewed by:

Copyright © 2024 Meng, Yu, Liu, Pan, Pang and Zhu. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Lei Yu, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Capturing the benefits of remote learning

How education experts are applying lessons learned in the pandemic to promote positive outcomes for all students

Vol. 52 No. 6 Print version: page 46

• Schools and Classrooms

With schools open again after more than a year of teaching students outside the classroom, the pandemic sometimes feels like a distant memory. The return to classrooms this fall brings major relief for many families and educators. Factors such as a lack of reliable technology and family support, along with an absence of school resources, resulted in significant academic setbacks, not to mention stress for everyone involved.

But for all the downsides of distance learning, educators, psychologists, and parents have seen some benefits as well. For example, certain populations of students found new ways to be more engaged in learning, without the distractions and difficulties they faced in the classroom, and the general challenges of remote learning and the pandemic brought mental health to the forefront of the classroom experience.

Peter Faustino, PsyD, a school psychologist in Scarsdale, New York, said the pandemic also prompted educators and school psychologists to find creative new ways of ensuring students’ emotional and academic well-being. “So many students were impacted by the pandemic, so we couldn’t just assume they would find resources on their own,” said Faustino. “We had to work hard at figuring out new ways to connect with them.”

Here are some of the benefits of distance learning that school psychologists and educators have observed and the ways in which they’re implementing those lessons post-pandemic, with the goal of creating a more equitable, productive environment for all students.

Prioritizing mental health

Faustino said that during the pandemic, he had more mental health conversations with students, families, and teachers than ever. “Because COVID-19 affected everyone, we’re now having mental health discussions as school leaders on a daily and weekly basis,” he said.

This renewed focus on mental health has the potential to improve students’ well-being in profound ways—starting with helping them recover from the pandemic’s effects. In New York City, for example, schools are hiring more than 600 new clinicians, including psychologists , to screen students’ mental health and help them process pandemic-related trauma and adjust to the “new normal” of attending school in person.

Educators and families are also realizing the importance of protecting students’ mental health more generally—not only for their health and safety but for their learning. “We’ve been seeing a broader appreciation for the fact that mental health is a prerequisite for learning rather than an extracurricular pursuit,” said Eric Rossen, PhD, director of professional development and standards at the National Association of School Psychologists.

As a result, Rossen hopes educators will embed social and emotional learning components into daily instruction. For example, teachers could teach mindfulness techniques in the classroom and take in-the-moment opportunities to help kids resolve conflicts or manage stress.

Improved access to mental health resources in schools is another positive effect. Because of physical distancing guidelines, school leaders had to find ways to deliver mental health services remotely, including via online referrals and teletherapy with school psychologists and counselors.

Early in the pandemic, Faustino said he was hesitant about teletherapy’s effectiveness; now, he hopes to continue offering a virtual option. Online scheduling and remote appointments make it easier for students to access mental health resources, and some students even enjoy virtual appointments more, as they can attend therapy in their own spaces rather than showing up in the counselor’s office. For older students, Faustino said that level of comfort often leads to more productive, open conversations.

Autonomy as a key to motivation

Research suggests that when students have more choices about their materials and activities, they’re more motivated—which may translate to increased learning and academic success. In a 2016 paper, psychology researcher Allan Wigfield, PhD, and colleagues make the case that control and autonomy in reading activities can improve both motivation and comprehension ( Child Development Perspectives , Vol. 10, No. 3 ).

During the period of online teaching, some students had opportunities to learn at their own pace, which educators say improved their learning outcomes—especially in older students. In a 2020 survey of more than 600 parents, researchers found the second-most-valued benefit of distance learning was flexibility—not only in schedule but in method of learning.

In a recent study, researchers found that 18% of parents pointed to greater flexibility in a child’s schedule or way of learning as the biggest benefit or positive outcome related to remote learning ( School Psychology , Roy, A., et al., in press).

This individualized learning helps students find more free time for interests and also allows them to conduct their learning at a time they’re most likely to succeed. During the pandemic, Mark Gardner, an English teacher at Hayes Freedom High School in Camas, Washington, said he realized how important student-centered learning is and that whether learning happens should take precedence over how and when it occurs.

For example, one of his students thrived when he had the choice to do work later at night because he took care of his siblings during the day. Now, Gardner posts homework online on Sundays so students can work at their own pace during the week. “Going forward, we want to create as many access points as we can for kids to engage with learning,” he said.

Rosanna Breaux , PhD, an assistant professor of psychology and assistant director of the Child Study Center at Virginia Tech, agrees. “I’d like to see this flexibility continue in some way, where—similar to college—students can guide their own learning based on their interests or when they’re most productive,” she said.

During the pandemic, many educators were forced to rethink how to keep students engaged. Rossen said because many school districts shared virtual curricula during the period of remote learning, older students could take more challenging or interesting courses than they could in person. The same is true for younger students: Megan Hibbard, a teacher in White Bear Lake, Minnesota, said many of her fifth graders enjoyed distance learning more than in-person because they could work on projects that aligned with their interests.

“So much of motivation is discovering the unique things the student finds interesting,” said Hunter Gehlbach, PhD, a professor and vice dean at the Johns Hopkins School of Education. “The more you can facilitate students spending more time on the things they’re really interested in, the better.”

Going forward, Rossen hopes virtual curricula will allow students greater opportunities to pursue their interests, such as by taking AP classes, foreign languages, or vocational electives not available at their own schools.

Conversely, Hibbard’s goal is to increase opportunities for students to pursue their interests in the in-person setting. For example, she plans to increase what she calls “Genius Hours,” a time at the end of the school day when students can focus on high-interest projects they’ll eventually share with the class.

Better understanding of children's needs

One of the most important predictors of a child’s success in school is parental involvement in their education. For example, in a meta-analysis of studies, researchers linked parental engagement in their middle schoolers’ education with greater measures of success (Hill, N. E., & Tyson, D. F., Developmental Psychology , Vol. 45, No. 3, 2009).

During the pandemic, parents had new opportunities to learn about their kids and, as a result, help them learn. According to a study by Breaux and colleagues, many parents reported that the pandemic allowed them a better understanding of their child’s learning style, needs, or curriculum.

James C. Kaufman , PhD, a professor of educational psychology at the University of Connecticut and the father of an elementary schooler and a high schooler, said he’s had a front-row seat for his sons’ learning for the first time. “Watching my kids learn and engage with classmates has given me some insight in how to parent them,” he said.

Stephen Becker , PhD, a pediatric psychologist at Cincinnati Children’s Hospital Medical Center, said some parents have observed their children’s behavior or learning needs for the first time, which could prompt them to consider assessment and Individualized Education Program (IEP) services. Across the board, Gehlbach said parents are realizing how they can better partner with schools to ensure their kids’ well-being and academic success.

For example, Samantha Marks , PsyD, a Florida-based clinical psychologist, said she realized how much help her middle school daughter, a gifted and talented student with a 504 plan (a plan for how the school will offer support for a student’s disability) for anxiety, needed with independence. “Bringing the learning home made it crystal clear what we needed to teach our daughter to be independent and improve executive functioning” she said. “My takeaway from this is that more parents need to be involved in their children’s education in a healthy, helpful way.”

Marks also gained a deeper understanding of her daughter’s mental health needs. Through her 504 plan, she received help managing her anxiety at school—at home, though, Marks wasn’t always available to help, which taught her the importance of helping her daughter manage her anxiety independently.

Along with parents gaining a deeper understanding of their kids’ needs, the pandemic also prompted greater parent participation in school. For example, Rossen said his kids’ school had virtual school board meetings; he hopes virtual options continue for events like back-to-school information sessions and parenting workshops. “These meetings are often in the evening, and if you’re a single parent or sole caregiver, you may not want to pay a babysitter in order to attend,” he said.

Brittany Greiert, PhD, a school psychologist in Aurora, Colorado, says culturally and linguistically diverse families at her schools benefited from streamlined opportunities to communicate with administrators and teachers. Her district used an app that translates parent communication into 150 languages. Parents can also remotely participate in meetings with school psychologists or teachers, which Greiert says she plans to continue post-pandemic.

Decreased bullying

During stay-at-home orders, kids with neurodevelopmental disorders experienced less bullying than pre-pandemic (McFayden, T. C., et al., Journal of Rural Mental Health , No. 45, Vol. 2, 2021). According to 2019 research, children with emotional, behavioral, and physical health needs experience increased rates of bullying victimization ( Lebrun-Harris, L. A., et al., ), and from the U.S. Department of Education suggests the majority of bullying takes place in person and in unsupervised areas (PDF) .

Scott Graves , PhD, an associate professor of educational studies at The Ohio State University and a member of APA’s Coalition for Psychology in Schools and Education (CPSE), said the supervision by parents and teachers in remote learning likely played a part in reducing bullying. As a result, he’s less worried his Black sons will be victims of microaggressions and racist behavior during online learning.

Some Asian American families also report that remote learning offered protection against racism students may have experienced in person. Shereen Naser, PhD, an associate professor of psychology at Cleveland State University and a member of CPSE, and colleagues found that students are more comfortable saying discriminatory things in school when their teachers are also doing so; Naser suspects this trickle-down effect is less likely to happen when students learn from home ( School Psychology International , 2019).

Reductions in bullying and microaggressions aren’t just beneficial for students’ long-term mental health. Breaux said less bullying at school results in less stress, which can improve students’ self-esteem and mood—both of which impact their ability to learn.

Patricia Perez, PhD, an associate professor of international psychology at The Chicago School of Professional Psychology and a member of CPSE, said it’s important for schools to be proactive in providing spaces for support and cultural expression for students from vulnerable backgrounds, whether in culture-specific clubs, all-school assemblies that address racism and other diversity-related topics, or safe spaces to process feelings with teachers.

According to Rossen, many schools are already considering how to continue supporting students at risk for bullying, including by restructuring the school environment.

One principal, Rossen said, recently switched to single-use bathrooms to avoid congregating in those spaces once in-person learning commences to maintain social distancing requirements. “The principal received feedback from students about how going to the bathroom is much less stressful for these students in part due to less bullying,” he said.

More opportunities for special needs students

In Becker and Breaux’s research, parents of students with attention-deficit/hyperactivity disorder (ADHD), particularly those with a 504 plan and IEP, reported greater difficulties with remote learning. But some students with special learning needs—including those with IEPs and 504 plans—thrived in an at-home learning environment. Recent reporting in The New York Times suggests this is one reason many students want to continue online learning.

According to Cara Laitusis, PhD, a principal research scientist at Educational Testing Service ( ETS ) and a member of CPSE, reduced distractions may improve learning outcomes for some students with disabilities that impact attention in a group setting. “In assessments, small group or individual settings are frequently requested accommodations for some students with ADHD, anxiety, or autism. Being in a quiet place alone without peers for part of the instructional day may also allow for more focus,” she said. However, she also pointed out the benefits of inclusion in the classroom for developing social skills with peers.

Remote learning has improved academic outcomes for students with different learning needs, too. Marks said her seventh-grade daughter, a visual learner, appreciated the increase in video presentations and graphics. Similarly, Hibbard said many of her students who struggle to grasp lessons on the first try have benefited from the ability to watch videos over again until they understand. Post-pandemic, she plans to record bite-size lessons—for example, a 1-minute video of a long division problem—so her students can rewatch and process at their own rate.

Learners with anxiety also appreciate the option not to be in the classroom, because the social pressures of being surrounded by peers can make it hard to focus on academics. “Several of my students have learned more in the last year simply due to the absence of anxiety,” said Rosie Reid, an English teacher at Ygnacio Valley High School in Concord, California, and a 2019 California Teacher of the Year. “It’s just one less thing to negotiate in a learning environment.”

On online learning platforms, it’s easier for kids with social anxiety or shyness to participate. One of Gardner’s students with social anxiety participated far more in virtual settings and chats. Now, Gardner is brainstorming ways to encourage students to chat in person, such as by projecting a chat screen on the blackboard.

Technology has helped school psychologists better engage students, too. For example, Greiert said the virtual setting gave her a new understanding of her students’ personalities and needs. “Typing out their thoughts, they were able to demonstrate humor or complex thoughts they never demonstrated in person,” she said. “I really want to keep incorporating technology into sessions so kids can keep building on their strengths.”

Reid says that along with the high school students she teaches, she’s seen her 6-year-old daughter benefit from learning at her own pace in the familiarity of her home. Before the pandemic, she was behind academically, but by guiding her own learning—writing poems, reading books, playing outside with her siblings—she’s blossomed. “For me, as both a mother and as a teacher, this whole phenomenon has opened the door to what education can be,” Reid said.

Eleanor Di Marino-Linnen, PhD, a psychologist and superintendent of the Rose Tree Media School District in Media, Pennsylvania, says the pandemic afforded her district a chance to rethink old routines and implement new ones. “As challenging as it is, it’s definitely an exciting time to be in education when we have a chance to reenvision what schools have looked like for many years,” she said. “We want to capitalize on what we’ve learned.”

Further reading

Why are some kids thriving during remote learning? Fleming, N., Edutopia, 2020

Remote learning has been a disaster for many students. But some kids have thrived. Gilman, A., The Washington Post , Oct. 3, 2020

A preliminary examination of key strategies, challenges, and benefits of remote learning expressed by parents during the COVID-19 pandemic Roy, A., et al., School Psychology , in press

Remote learning during COVID-19: Examining school practices, service continuation, and difficulties for adolescents with and without attention-deficit/hyperactivity disorder Becker S. P., et al., Journal of Adolescent Health , 2020

Recommended Reading

Contact APA

You may also like.

• Open access
• Published: 16 September 2021

Online learning during COVID-19 produced equivalent or better student course performance as compared with pre-pandemic: empirical evidence from a school-wide comparative study

• Meixun Zheng 1 ,
• Daniel Bender 1 &
• Cindy Lyon 1  

BMC Medical Education volume  21 , Article number:  495 ( 2021 ) Cite this article

214k Accesses

90 Citations

115 Altmetric

Metrics details

The COVID-19 pandemic forced dental schools to close their campuses and move didactic instruction online. The abrupt transition to online learning, however, has raised several issues that have not been resolved. While several studies have investigated dental students’ attitude towards online learning during the pandemic, mixed results have been reported. Additionally, little research has been conducted to identify and understand factors, especially pedagogical factors, that impacted students’ acceptance of online learning during campus closure. Furthermore, how online learning during the pandemic impacted students’ learning performance has not been empirically investigated. In March 2020, the dental school studied here moved didactic instruction online in response to government issued stay-at-home orders. This first-of-its-kind comparative study examined students’ perceived effectiveness of online courses during summer quarter 2020, explored pedagogical factors impacting their acceptance of online courses, and empirically evaluated the impact of online learning on students’ course performance, during the pandemic.

The study employed a quasi-experimental design. Participants were 482 pre-doctoral students in a U.S dental school. Students’ perceived effectiveness of online courses during the pandemic was assessed with a survey. Students’ course grades for online courses during summer quarter 2020 were compared with that of a control group who received face-to-face instruction for the same courses before the pandemic in summer quarter 2019.

Survey results revealed that most online courses were well accepted by the students, and 80 % of them wanted to continue with some online instruction post pandemic. Regression analyses revealed that students’ perceived engagement with faculty and classmates predicted their perceived effectiveness of the online course. More notably, Chi Square tests demonstrated that in 16 out of the 17 courses compared, the online cohort during summer quarter 2020 was equally or more likely to get an A course grade than the analogous face-to-face cohort during summer quarter 2019.

Conclusions

This is the first empirical study in dental education to demonstrate that online courses during the pandemic could achieve equivalent or better student course performance than the same pre-pandemic in-person courses. The findings fill in gaps in literature and may inform online learning design moving forward.

Peer Review reports

Introduction

Research across disciplines has demonstrated that well-designed online learning can lead to students’ enhanced motivation, satisfaction, and learning [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ]. A report by the U.S. Department of Education [ 8 ], based on examinations of comparative studies of online and face-to-face versions of the same course from 1996 to 2008, concluded that online learning could produce learning outcomes equivalent to or better than face-to-face learning. The more recent systematic review by Pei and Wu [ 9 ] provided additional evidence that online learning is at least as effective as face-to-face learning for undergraduate medical students.

To take advantage of the opportunities presented by online learning, thought leaders in dental education in the U.S. have advocated for the adoption of online learning in the nation’s dental schools [ 10 , 11 , 12 ]. However, digital innovation has been a slow process in academic dentistry [ 13 , 14 , 15 ]. In March 2020, the COVID-19 pandemic brought unprecedented disruption to dental education by necessitating the need for online learning. In accordance with stay-at-home orders to prevent the spread of the virus, dental schools around the world closed their campuses and moved didactic instruction online.

The abrupt transition to online learning, however, has raised several concerns and question. First, while several studies have examined dental students’ online learning satisfaction during the pandemic, mixed results have been reported. Some studies have reported students’ positive attitude towards online learning [ 15 , 16 , 17 , 18 , 19 , 20 ]. Sadid-Zadeh et al. [ 18 ] found that 99 % of the surveyed dental students at University of Buffalo, in the U.S., were satisfied with live web-based lectures during the pandemic. Schlenz et al. [ 15 ] reported that students in a German dental school had a favorable attitude towards online learning and wanted to continue with online instruction in their future curriculum. Other studies, however, have reported students’ negative online learning experience during the pandemic [ 21 , 22 , 23 , 24 , 25 , 26 ]. For instance, dental students at Harvard University felt that learning during the pandemic had worsened and engagement had decreased [ 23 , 24 ]. In a study with medical and dental students in Pakistan, Abbasi et al. [ 21 ] found that 77 % of the students had negative perceptions about online learning and 84 % reported reduced student-instructor interactions.

In addition to these mixed results, little attention has been given to factors affecting students’ acceptance of online learning during the pandemic. With the likelihood that online learning will persist post pandemic [ 27 ], research in this area is warranted to inform online course design moving forward. In particular, prior research has demonstrated that one of the most important factors influencing students’ performance in any learning environment is a sense of belonging, the feeling of being connected with and supported by the instructor and classmates [ 28 , 29 , 30 , 31 ]. Unfortunately, this aspect of the classroom experience has suffered during school closure. While educational events can be held using a video conferencing system, virtual peer interaction on such platforms has been perceived by medical trainees to be not as easy and personal as physical interaction [ 32 ]. The pandemic highlights the need to examine instructional strategies most suited to the current situation to support students’ engagement with faculty and classmates.

Furthermore, there is considerable concern from the academic community about the quality of online learning. Pre-pandemic, some faculty and students were already skeptical about the value of online learning [ 33 ]. The longer the pandemic lasts, the more they may question the value of online education, asking: Can online learning during the pandemic produce learning outcomes that are similar to face-to-face learning before the pandemic? Despite the documented benefits of online learning prior to the pandemic, the actual impact of online learning during the pandemic on students’ academic performance is still unknown due to reasons outlined below.

On one hand, several factors beyond the technology used could influence the effectiveness of online learning, one of which is the teaching context [ 34 ]. The sudden transition to online learning has posed many challenges to faculty and students. Faculty may not have had adequate time to carefully design online courses to take full advantage of the possibilities of the online format. Some faculty may not have had prior online teaching experience and experienced a deeper learning curve when it came to adopting online teaching methods [ 35 ]. Students may have been at the risk of increased anxiety due to concerns about contracting the virus, on time graduation, finances, and employment [ 36 , 37 ], which may have negatively impacted learning performance [ 38 ]. Therefore, whether online learning during the pandemic could produce learning outcomes similar to those of online learning implemented during more normal times remains to be determined.

Most existing studies on online learning in dental education during the pandemic have only reported students’ satisfaction. The actual impact of the online format on academic performance has not been empirically investigated. The few studies that have examined students’ learning outcomes have only used students’ self-reported data from surveys and focus groups. According to Kaczmarek et al. [ 24 ], 50 % of the participating dental faculty at Harvard University perceived student learning to have worsened during the pandemic and 70 % of the students felt the same. Abbasi et al. [ 21 ] reported that 86 % of medical and dental students in a Pakistan college felt that they learned less online. While student opinions are important, research has demonstrated a poor correlation between students’ perceived learning and actual learning gains [ 39 ]. As we continue to navigate the “new normal” in teaching, students’ learning performance needs to be empirically evaluated to help institutions gauge the impact of this grand online learning experiment.

Research purposes

In March 2020, the University of the Pacific Arthur A. Dugoni School of Dentistry, in the U.S., moved didactic instruction online to ensure the continuity of education during building closure. This study examined students’ acceptance of online learning during the pandemic and its impacting factors, focusing on instructional practices pertaining to students’ engagement/interaction with faculty and classmates. Another purpose of this study was to empirically evaluate the impact of online learning during the pandemic on students’ actual course performance by comparing it with that of a pre-pandemic cohort. To understand the broader impact of the institutional-wide online learning effort, we examined all online courses offered in summer quarter 2020 (July to September) that had a didactic component.

This is the first empirical study in dental education to evaluate students’ learning performance during the pandemic. The study aimed to answer the following three questions.

How well was online learning accepted by students, during the summer quarter 2020 pandemic interruption?

How did instructional strategies, centered around students’ engagement with faculty and classmates, impact their acceptance of online learning?

How did online learning during summer quarter 2020 impact students’ course performance as compared with a previous analogous cohort who received face-to-face instruction in summer quarter 2019?

This study employed a quasi-experimental design. The study was approved by the university’s institutional review board (#2020-68).

Study context and participants

The study was conducted at the Arthur A. Dugoni School of Dentistry, University of the Pacific. The program runs on a quarter system. It offers a 3-year accelerated Doctor of Dental Surgery (DDS) program and a 2-year International Dental Studies (IDS) program for international dentists who have obtained a doctoral degree in dentistry from a country outside the U.S. and want to practice in the U.S. Students advance throughout the program in cohorts. IDS students take some courses together with their DDS peers. All three DDS classes (D1/DDS 2023, D2/DDS 2022, and D3/DDS 2021) and both IDS classes (I1/IDS 2022 and I2/IDS 2021) were invited to participate in the study. The number of students in each class was: D1 = 145, D2 = 143, D3 = 143, I1 = 26, and I2 = 25. This resulted in a total of 482 student participants.

During campus closure, faculty delivered remote instruction in various ways, including live online classes via Zoom @  [ 40 ], self-paced online modules on the school’s learning management system Canvas @  [ 41 ], or a combination of live and self-paced delivery. For self-paced modules, students studied assigned readings and/or viewings such as videos and pre-recorded slide presentations. Some faculty also developed self-paced online lessons with SoftChalk @  [ 42 ], a cloud-based platform that supports the inclusion of gamified learning by insertion of various mini learning activities. The SoftChalk lessons were integrated with Canvas @  [ 41 ] and faculty could monitor students’ progress. After students completed the pre-assigned online materials, some faculty held virtual office hours or live online discussion sessions for students to ask questions and discuss key concepts.

Data collection and analysis

Student survey.

Students’ perceived effectiveness of summer quarter 2020 online courses was evaluated by the school’s Office of Academic Affairs in lieu of the regular course evaluation process. A total of 19 courses for DDS students and 10 courses for IDS students were evaluated. An 8-question survey developed by the researchers (Additional file 1 ) was administered online in the last week of summer quarter 2020. Course directors invited student to take the survey during live online classes. The survey introduction stated that taking the survey was voluntary and that their anonymous responses would be reported in aggregated form for research purposes. Students were invited to continue with the survey if they chose to participate; otherwise, they could exit the survey. The number of students in each class who took the survey was as follows: D1 ( n  = 142; 98 %), D2 ( n  = 133; 93 %), D3 ( n  = 61; 43 %), I1 ( n  = 23; 88 %), and I2 ( n  = 20; 80 %). This resulted in a total of 379 (79 %) respondents across all classes.

The survey questions were on a 4-point scale, ranging from Strongly Disagree (1 point), Disagree (2 points), Agree (3 points), and Strongly Agree (4 points). Students were asked to rate each online course by responding to four statements: “ I could fully engage with the instructor and classmates in this course”; “The online format of this course supported my learning”; “Overall this online course is effective.”, and “ I would have preferred face-to-face instruction for this course ”. For the first three survey questions, a higher mean score indicated a more positive attitude toward the online course. For the fourth question “ I would have preferred face-to-face instruction for this course ”, a higher mean score indicated that more students would have preferred face-to-face instruction for the course. Two additional survey questions asked students to select their preferred online delivery method for fully online courses during the pandemic from three given choices (synchronous online/live, asynchronous online/self-paced, and a combination of both), and to report whether they wanted to continue with some online instruction post pandemic. Finally, two open-ended questions at the end of the survey allowed students to comment on the aspects of online format that they found to be helpful and to provide suggestion for improvement. For the purpose of this study, we focused on the quantitative data from the Likert-scale questions.

Descriptive data such as the mean scores were reported for each course. Regression analyses were conducted to examine the relationship between instructional strategies focusing on students’ engagement with faculty and classmates, and their overall perceived effectiveness of the online course. The independent variable was student responses to the question “ I could fully engage with the instructor and classmates in this course ”, and the dependent variable was their answer to the question “ Overall, this online course is effective .”

Student course grades

Using Chi-square tests, student course grade distributions (A, B, C, D, and F) for summer quarter 2020 online courses were compared with that of a previous cohort who received face-to-face instruction for the same course in summer quarter 2019. Note that as a result of the school’s pre-doctoral curriculum redesign implemented in July 2019, not all courses offered in summer quarter 2020 were offered in the previous year in summer quarter 2019. In other words, some of the courses offered in summer quarter 2020 were new courses offered for the first time. Because these new courses did not have a previous face-to-face version to compare to, they were excluded from data analysis. For some other courses, while course content remained the same between 2019 and 2020, the sequence of course topics within the course had changed. These courses were also excluded from data analysis.

After excluding the aforementioned courses, it resulted in a total of 17 “comparable” courses that were included in data analysis (see the subsequent section). For these courses, the instructor, course content, and course goals were the same in both 2019 and 2020. The assessment methods and grading policies also remained the same through both years. For exams and quizzes, multiple choice questions were the dominating format for both years. While some exam questions in 2020 were different from 2019, faculty reported that the overall exam difficulty level was similar. The main difference in assessment was testing conditions. The 2019 cohort took computer-based exams in the physical classroom with faculty proctoring, and the 2020 cohort took exams at home with remote proctoring to ensure exam integrity. The remote proctoring software monitored the student during the exam through a web camera on their computer/laptop. The recorded video file flags suspicious activities for faculty review after exam completion.

Students’ perceived effectiveness of online learning

Table  1 summarized data on DDS students’ perceived effectiveness of each online course during summer quarter 2020. For the survey question “ Overall, this online course is effective ”, the majority of courses received a mean score that was approaching or over 3 points on the 4-point scale, suggesting that online learning was generally well accepted by students. Despite overall positive online course experiences, for many of the courses examined, there was an equal split in student responses to the question “ I would have preferred face-to-face instruction for this course .” Additionally, for students’ preferred online delivery method for fully online courses, about half of the students in each class preferred a combination of synchronous and asynchronous online learning (see Fig.  1 ). Finally, the majority of students wanted faculty to continue with some online instruction post pandemic: D1class (110; 78.60 %), D2 class (104; 80 %), and D3 class (49; 83.10 %).

While most online courses received favorable ratings, some variations did exist among courses. For D1 courses, “ Anatomy & Histology ” received lower ratings than others. This could be explained by its lab component, which didn’t lend itself as well to the online format. For D2 courses, several of them received lower ratings than others, especially for the survey question on students’ perceived engagement with faculty and classmates.

DDS students’ preferred online delivery method for fully online courses

Table  2 summarized IDS students’ perceived effectiveness of each online course during summer quarter 2020. For the survey question “ Overall, this online course is effective ”, all courses received a mean score that was approaching or over 3 points on a 4-point scale, suggesting that online learning was well accepted by students. For the survey question “ I would have preferred face-to-face instruction for this course ”, for most online courses examined, the percentage of students who would have preferred face-to-face instruction was similar to that of students who preferred online instruction for the course. Like their DDS peers, about half of the IDS students in each class also preferred a combination of synchronous and asynchronous online delivery for fully online courses (See Fig.  2 ). Finally, the majority of IDS students (I1, n = 18, 81.80 %; I2, n = 16, 84.20 %) wanted to continue with some online learning after the pandemic is over.

IDS students’ preferred online delivery method for fully online courses

Factors impacting students’ acceptance of online learning

For all 19 online courses taken by DDS students, regression analyses indicated that there was a significantly positive relationship between students’ perceived engagement with faculty and classmates and their perceived effectiveness of the course. P value was 0.00 across all courses. The ranges of effect size (r 2 ) were: D1 courses (0.26 to 0.50), D2 courses (0.39 to 0.650), and D3 courses (0.22 to 0.44), indicating moderate to high correlations across courses.

For 9 out of the 10 online courses taken by IDS students, there was a positive relationship between students’ perceived engagement with faculty and classmates and their perceived effectiveness of the course. P value was 0.00 across courses. The ranges of effect size were: I1 courses (0.35 to 0.77) and I2 courses (0.47 to 0.63), indicating consistently high correlations across courses. The only course in which students’ perceived engagement with faculty and classmates didn’t predict perceived effective of the course was “ Integrated Clinical Science III (ICS III) ”, which the I2 class took together with their D3 peers.

Impact of online learning on students’ course performance

Chi square test results (Table  3 ) indicated that in 4 out of the 17 courses compared, the online cohort during summer quarter 2020 was more likely to receive an A grade than the face-to-face cohort during summer quarter 2019. In 12 of the courses, the online cohort were equally likely to receive an A grade as the face-to-face cohort. In the remaining one course, the online cohort was less likely to receive an A grade than the face-to-face cohort.

Students’ acceptance of online learning during the pandemic

Survey results revealed that students had generally positive perceptions about online learning during the pandemic and the majority of them wanted to continue with some online learning post pandemic. Overall, our findings supported several other studies in dental [ 18 , 20 ], medical [ 43 , 44 ], and nursing [ 45 ] education that have also reported students’ positive attitudes towards online learning during the pandemic. In their written comments in the survey, students cited enhanced flexibility as one of the greatest benefits of online learning. Some students also commented that typing questions in the chat box during live online classes was less intimidating than speaking in class. Others explicitly stated that not having to commute to/from school provided more time for sleep, which helped with self-care and mental health. Our findings are in line with previous studies which have also demonstrated that online learning offered higher flexibility [ 46 , 47 ]. Meanwhile, consistent with findings of other researchers [ 19 , 21 , 46 ], our students felt difficulty engaging with faculty and classmates in several online courses.

There were some variations among individual courses in students’ acceptance of the online format. One factor that could partially account for the observed differences was instructional strategies. In particular, our regression analysis results demonstrated a positive correlation between students’ perceived engagement with faculty and classmates and their perceived overall effectiveness of the online course. Other aspects of course design might also have influenced students’ overall rating of the online course. For instance, some D2 students commented that the requirements of the course “ Integrated Case-based Seminars (ICS II) ” were not clear and that assessment did not align with lecture materials. It is important to remember that communicating course requirements clearly and aligning course content and assessment are principles that should be applied in any course, whether face-to-face or online. Our results highlighted the importance of providing faculty training on basic educational design principles and online learning design strategies. Furthermore, the nature of the course might also have impacted student ratings. For example, D1 course “ Anatomy and Histology ” had a lab component, which did not lend itself as well to the online format. Many students reported that it was difficult to see faculty’s live demonstration during Zoom lectures, which may have resulted in a lower student satisfaction rating.

As for students’ preferred online delivery method for fully online courses during the pandemic, about half of them preferred a combination of synchronous and asynchronous online learning. In light of this finding, as we continue with remote learning until public health directives allow a return to campus, we will encourage faculty to integrate these two online delivery modalities. Finally, in view of the result that over 80 % of the students wanted to continue with some online instruction after the pandemic, the school will advocate for blended learning in the post-pandemic world [ 48 ]. For future face-to-face courses on campus after the pandemic, faculty are encouraged to deliver some content online to reduce classroom seat time and make learning more flexible. Taken together, our findings not only add to the overall picture of the current situation but may inform learning design moving forward.

Role of online engagement and interaction

To reiterate, we found that students’ perceived engagement with faculty and classmates predicted their perceived overall effectiveness of the online course. This aligns with the larger literature on best practices in online learning design. Extensive research prior to the pandemic has confirmed that the effectiveness of online learning is determined by a number of factors beyond the tools used, including students’ interactions with the instructor and classmates [ 49 , 50 , 51 , 52 ]. Online students may feel isolated due to reduced or lack of interaction [ 53 , 54 ]. Therefore, in designing online learning experiences, it is important to remember that learning is a social process [ 55 ]. Faculty’s role is not only to transmit content but also to promote the different types of interactions that are an integral part of the online learning process [ 33 ]. The online teaching model in which faculty uploads materials online but teach it in the same way as in the physical classroom, without special effort to engage students, doesn’t make the best use of the online format. Putting the “sage on the screen” during a live class meeting on a video conferencing system is not different from “sage on the stage” in the physical classroom - both provide limited space for engagement. Such one-way monologue devalues the potentials that online learning presents.

In light of the critical role that social interaction plays in online learning, faculty are encouraged to use the interactive features of online learning platforms to provide clear channels for student-instructor and student-student interactions. In the open-ended comments, students highlighted several instructional strategies that they perceived to be helpful for learning. For live online classes, these included conducting breakout room activities, using the chat box to facilitate discussions, polling, and integrating gameplay with apps such as Kahoot! @  [ 56 ]. For self-paced classes, students appreciated that faculty held virtual office hours or subsequent live online discussion sessions to reinforce understanding of the pre-assigned materials.

Quality of online education during the pandemic

This study provided empirical evidence in dental education that it was possible to ensure the continuity of education without sacrificing the quality of education provided to students during forced migration to distance learning upon building closure. To reiterate, in all but one online course offered in summer quarter 2020, students were equally or more likely to get an A grade than the face-to-face cohort from summer quarter 2019. Even for courses that had less student support for the online format (e.g., the D1 course “ Anatomy and Histology ”), there was a significant increase in the number of students who earned an A grade in 2020 as compared with the previous year. The reduced capacity for technical training during the pandemic may have resulted in more study time for didactic content. Overall, our results resonate with several studies in health sciences education before the pandemic that the quality of learning is comparable in face-to-face and online formats [ 9 , 57 , 58 ]. For the only course ( Integrated Case-based Seminars ICS II) in which the online cohort had inferior performance than the face-to-face cohort, as mentioned earlier, students reported that assessment was not aligned with course materials and that course expectations were not clear. This might explain why students’ course performance was not as strong as expected.

Limitations

This study used a pre-existing control group from the previous year. There may have been individual differences between students in the online and the face-to-face cohorts, such as motivation, learning style, and prior knowledge, that could have impacted the observed outcomes. Additionally, even though course content and assessment methods were largely the same in 2019 and 2020, changes in other aspects of the course could have impacted students’ course performance. Some faculty may have been more compassionate with grading (e.g., more flexible with assignment deadlines) in summer quarter 2020 given the hardship students experienced during the pandemic. On the other hand, remote proctoring in summer quarter 2020 may have heightened some students’ exam anxiety knowing that they were being monitored through a webcam. The existence and magnitude of effect of these factors needs to be further investigated.

This present study only examined the correlation between students’ perceived online engagement and their perceived overall effectiveness of the online course. Other factors that might impact their acceptance of the online format need to be further researched in future studies. Another future direction is to examine how students’ perceived online engagement correlates with their actual course performance. Because the survey data collected for our present study are anonymous, we cannot match students’ perceived online engagement data with their course grades to run this additional analysis. It should also be noted that this study was focused on didactic online instruction. Future studies might examine how technical training was impacted during the COVID building closure. It was also out of the scope of this study to examine how student characteristics, especially high and low academic performance as reflected by individual grades, affects their online learning experience and performance. We plan to conduct a follow-up study to examine which group of students are most impacted by the online format. Finally, this study was conducted in a single dental school, and so the findings may not be generalizable to other schools and disciplines. Future studies could be conducted in another school or disciplines to compare results.

This study revealed that dental students had generally favorable attitudes towards online learning during the COVID-19 pandemic and that their perceived engagement with faculty and classmates predicted their acceptance of the online course. Most notably, this is the first study in dental education to demonstrate that online learning during the pandemic could achieve similar or better learning outcomes than face-to-face learning before the pandemic. Findings of our study could contribute significantly to the literature on online learning during the COVID-19 pandemic in health sciences education. The results could also inform future online learning design as we re-envision the future of online learning.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Bello G, Pennisi MA, Maviglia R, Maggiore SM, Bocci MG, Montini L, et al. Online vs live methods for teaching difficult airway management to anesthesiology residents. Intensive Care Med. 2005; 31 (4): 547–552.

Article   Google Scholar  

Ruiz JG, Mintzer MJ, Leipzig RM. The impact of e-learning in medical education. Acad Med. 2006; 81(3): 207–12.

Kavadella A, Tsiklakis K, Vougiouklakis G, Lionarakis A. Evaluation of a blended learning course for teaching oral radiology to undergraduate dental students. Eur J Dent Educ. 2012; 16(1): 88–95.

de Jong N, Verstegen DL, Tan FS, O’Connor SJ. A comparison of classroom and online asynchronous problem-based learning for students undertaking statistics training as part of a public health master’s degree. Adv Health Sci Educ. 2013; 18(2):245–64.

Hegeman JS. Using instructor-generated video lectures in online mathematics coursesimproves student learning. Online Learn. 2015;19(3):70–87.

Gaupp R, Körner M, Fabry G. Effects of a case-based interactive e-learning course on knowledge and attitudes about patient safety: a quasi-experimental study with third-year medical students. BMC Med Educ. 2016; 16(1):172.

Zheng M, Bender D, Reid L, Milani J. An interactive online approach to teaching evidence-based dentistry with Web 2.0 technology. J Dent Educ. 2017; 81(8): 995–1003.

Means B, Toyama Y, Murphy R, Bakia M, Jones K. Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. U.S. Department of Education, Office of Planning, Evaluation and Policy Development. Washington D.C. 2009.

Google Scholar  

Pei L, Wu H. Does online learning work better than offline learning in undergraduate medical education? A systematic review and meta-analysis. Med Educ Online. 2019; 24(1):1666538.

Andrews KG, Demps EL. Distance education in the U.S. and Canadian undergraduate dental curriculum. J Dent Educ. 2003; 67(4):427–38.

Kassebaum DK, Hendricson WD, Taft T, Haden NK. The dental curriculum at North American dental institutions in 2002–03: a survey of current structure, recent innovations, and planned changes. J Dent Educ. 2004; 68(9):914–931.

Haden NK, Hendricson WD, Kassebaum DK, Ranney RR, Weinstein G, Anderson EL, et al. Curriculum changes in dental education, 2003–09. J Dent Educ. 2010; 74(5):539–57.

DeBate RD, Cragun D, Severson HH, Shaw T, Christiansen S, Koerber A, et al. Factors for increasing adoption of e-courses among dental and dental hygiene faculty members. J Dent Educ. 2011; 75 (5): 589–597.

Saeed SG, Bain J, Khoo E, Siqueira WL. COVID-19: Finding silver linings for dental education. J Dent Educ. 2020; 84(10):1060–1063.

Schlenz MA, Schmidt A, Wöstmann B, Krämer N, Schulz-Weidner N. Students’ and lecturers’ perspective on the implementation of online learning in dental education due to SARS-CoV-2 (COVID-19): a cross-sectional study. BMC Med Educ. 2020;20(1):1–7.

Donn J, Scott JA, Binnie V, Bell A. A pilot of a virtual Objective Structured Clinical Examination in dental education. A response to COVID-19. Eur J Dent Educ. 2020; https://doi.org/10.1111/eje.12624

Hung M, Licari FW, Hon ES, Lauren E, Su S, Birmingham WC, Wadsworth LL, Lassetter JH, Graff TC, Harman W, et al. In an era of uncertainty: impact of COVID-19 on dental education. J Dent Educ. 2020; 85 (2): 148–156.

Sadid-Zadeh R, Wee A, Li R, Somogyi‐Ganss E. Audience and presenter comparison of live web‐based lectures and traditional classroom lectures during the COVID‐19 pandemic. J Prosthodont. 2020. doi: https://doi.org/10.1111/jopr.13301

Wang K, Zhang L, Ye L. A nationwide survey of online teaching strategies in dental education in China. J Dent Educ. 2020; 85 (2): 128–134.

Rad FA, Otaki F, Baqain Z, Zary N, Al-Halabi M. Rapid transition to distance learning due to COVID-19: Perceptions of postgraduate dental learners and instructors. PLoS One. 2021; 16(2): e0246584.

Abbasi S, Ayoob T, Malik A, Memon SI. Perceptions of students regarding E-learning during Covid-19 at a private medical college. Pak J Med Sci. 2020; 3 6 : 57–61.

Al-Azzam N, Elsalem L, Gombedza F. A cross-sectional study to determine factors affecting dental and medical students’ preference for virtual learning during the COVID-19 outbreak. Heliyon. 6(12). 2020. doi: https://doi.org/10.1016/j.heliyon.2020.e05704

Chen E, Kaczmarek K, Ohyama H. Student perceptions of distance learning strategies during COVID-19. J Dent Educ. 2020. doi: https://doi.org/10.1002/jdd.12339

Kaczmarek K, Chen E, Ohyama H. Distance learning in the COVID-19 era: Comparison of student and faculty perceptions. J Dent Educ. 2020. https://doi.org/10.1002/jdd.12469

Sarwar H, Akhtar H, Naeem MM, Khan JA, Waraich K, Shabbir S, et al. Self-reported effectiveness of e-learning classes during COVID-19 pandemic: A nation-wide survey of Pakistani undergraduate dentistry students. Eur J Dent. 2020; 14 (S01): S34-S43.

Al-Taweel FB, Abdulkareem AA, Gul SS, Alshami ML. Evaluation of technology‐based learning by dental students during the pandemic outbreak of coronavirus disease 2019. Eur J Dent Educ. 2021; 25(1): 183–190.

Elangovan S, Mahrous A, Marchini L. Disruptions during a pandemic: Gaps identified and lessons learned. J Dent Educ. 2020; 84 (11): 1270–1274.

Goodenow C. Classroom belonging among early adolescent students: Relationships to motivation and achievement. J Early Adolesc.1993; 13(1): 21–43.

Goodenow C. The psychological sense of school membership among adolescents: Scale development and educational correlates. Psychol Sch. 1993; 30(1): 79–90.

St-Amand J, Girard S, Smith J. Sense of belonging at school: Defining attributes, determinants, and sustaining strategies. IAFOR Journal of Education. 2017; 5(2):105–19.

Peacock S, Cowan J. Promoting sense of belonging in online learning communities of inquiry at accredited courses. Online Learn. 2019; 23(2): 67–81.

Chan GM, Kanneganti A, Yasin N, Ismail-Pratt I, Logan SJ. Well‐being, obstetrics and gynecology and COVID‐19: Leaving no trainee behind. Aust N Z J Obstet Gynaecol. 2020; 60(6): 983–986.

Hodges C, Moore S, Lockee B, Trust T, Bond A. The difference between emergency remote teaching and online learning. Educause Review. 2020; 2 7 , 1–12.

Means B, Bakia M, Murphy R. Learning online: What research tells us about whether, when and how. Routledge. 2014.

Iyer P, Aziz K, Ojcius DM. Impact of COVID-19 on dental education in the United States. J Dent Educ. 2020; 84(6): 718–722.

Machado RA, Bonan PRF, Perez DEDC, Martelli JÚnior H. 2020. COVID-19 pandemic and the impact on dental education: Discussing current and future perspectives. Braz Oral Res. 2020; 34: e083.

Wu DT, Wu KY, Nguyen TT, Tran SD. The impact of COVID-19 on dental education in North America-Where do we go next? Eur J Dent Educ. 2020; 24(4): 825–827.

de Oliveira Araújo FJ, de Lima LSA, Cidade PIM, Nobre CB, Neto MLR. Impact of Sars-Cov-2 and its reverberation in global higher education and mental health. Psychiatry Res. 2020; 288:112977. doi: https://doi.org/10.1016/j.psychres.2020.112977

Persky AM, Lee E, Schlesselman LS. Perception of learning versus performance as outcome measures of educational research. Am J Pharm Educ. 2020; 8 4 (7): ajpe7782.

Zoom @ . Zoom Video Communications , San Jose, CA, USA. https://zoom.us/

Canvas @ . Instructure, INC. Salt Lake City, UT, USA. https://www.instructure.com/canvas

SoftChalk @ . SoftChalk LLC . San Antonio, TX, USA. https://www.softchalkcloud.com/

Agarwal S, Kaushik JS. Student’s perception of online learning during COVID pandemic. Indian J Pediatr. 2020; 87: 554–554.

Khalil R, Mansour AE, Fadda WA, Almisnid K, Aldamegh M, Al-Nafeesah A, et al. The sudden transition to synchronized online learning during the COVID-19 pandemic in Saudi Arabia: a qualitative study exploring medical students’ perspectives. BMC Med Educ. 2020; 20(1): 1–10.

Riley E, Capps N, Ward N, McCormack L, Staley J. Maintaining academic performance and student satisfaction during the remote transition of a nursing obstetrics course to online instruction. Online Learn. 2021; 25(1), 220–229.

Amir LR, Tanti I, Maharani DA, Wimardhani YS, Julia V, Sulijaya B, et al. Student perspective of classroom and distance learning during COVID-19 pandemic in the undergraduate dental study program Universitas Indonesia. BMC Med Educ. 2020; 20(1):1–8.

Dost S, Hossain A, Shehab M, Abdelwahed A, Al-Nusair L. Perceptions of medical students towards online teaching during the COVID-19 pandemic: a national cross-sectional survey of 2721 UK medical students. BMJ Open. 2020; 10(11).

Graham CR, Woodfield W, Harrison JB. A framework for institutional adoption and implementation of blended learning in higher education. Internet High Educ. 2013; 18 : 4–14.

Sing C, Khine M. An analysis of interaction and participation patterns in online community. J Educ Techno Soc. 2006; 9(1): 250–261.

Bernard RM, Abrami PC, Borokhovski E, Wade CA, Tamim RM, Surkes MA, et al. A meta-analysis of three types of interaction treatments in distance education. Rev Educ Res. 2009; 79(3): 1243–1289.

Fedynich L, Bradley KS, Bradley J. Graduate students’ perceptions of online learning. Res High Educ. 2015; 27.

Tanis CJ. The seven principles of online learning: Feedback from faculty and alumni on its importance for teaching and learning. Res Learn Technol. 2020; 28 . https://doi.org/10.25304/rlt.v28.2319

Dixson MD. Measuring student engagement in the online course: The Online Student Engagement scale (OSE). Online Learn. 2015; 19 (4).

Kwary DA, Fauzie S. Students’ achievement and opinions on the implementation of e-learning for phonetics and phonology lectures at Airlangga University. Educ Pesqui. 2018; 44 .

Vygotsky LS. Mind in society: The development of higher psychological processes. Cambridge (MA): Harvard University Press. 1978.

Kahoot! @ . Oslo, Norway. https://kahoot.com/

Davis J, Chryssafidou E, Zamora J, Davies D, Khan K, Coomarasamy A. Computer-based teaching is as good as face to face lecture-based teaching of evidence-based medicine: a randomised controlled trial. BMC Med Educ. 2007; 7(1): 1–6.

Davis J, Crabb S, Rogers E, Zamora J, Khan K. Computer-based teaching is as good as face to face lecture-based teaching of evidence-based medicine: a randomized controlled trial. Med Teach. 2008; 30(3): 302–307.

Download references

Acknowledgements

Not applicable.

Authors’ information

MZ is an Associate Professor of Learning Sciences and Senior Instructional Designer at School of Dentistry, University of the Pacific. She has a PhD in Education, with a specialty on learning sciences and technology. She has dedicated her entire career to conducting research on online learning, learning technology, and faculty development. Her research has resulted in several peer-reviewed publications in medical, dental, and educational technology journals. MZ has also presented regularly at national conferences.

DB is an Assistant Dean for Academic Affairs at School of Dentistry, University of the Pacific. He has an EdD degree in education, with a concentration on learning and instruction. Over the past decades, DB has been overseeing and delivering faculty pedagogical development programs to dental faculty. His research interest lies in educational leadership and instructional innovation. DB has co-authored several peer-reviewed publications in health sciences education and presented regularly at national conferences.

CL is Associate Dean of Oral Healthcare Education, School of Dentistry, University of the Pacific. She has a Doctor of Dental Surgery (DDS) degree and an EdD degree with a focus on educational leadership. Her professional interest lies in educational leadership, oral healthcare education innovation, and faculty development. CL has co-authored several publications in peer-reviewed journals in health sciences education and presented regularly at national conferences.

Author information

Authors and affiliations.

Office of Academic Affairs, Arthur A. Dugoni School of Dentistry, University of the Pacific, CA, San Francisco, USA

Meixun Zheng, Daniel Bender & Cindy Lyon

You can also search for this author in PubMed   Google Scholar

Contributions

MZ analyzed the data and wrote the initial draft of the manuscript. DB and CL both provided assistance with research design, data collection, and reviewed and edited the manuscript. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Meixun Zheng .

Ethics declarations

Ethics approval and consent to participate.

The study was approved by the institutional review board at University of the Pacific in the U.S. (#2020-68). Informed consent was obtained from all participants. All methods were carried out in accordance with relevant guidelines and regulations.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:.

Survey of online courses during COVID-19 pandemic.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Zheng, M., Bender, D. & Lyon, C. Online learning during COVID-19 produced equivalent or better student course performance as compared with pre-pandemic: empirical evidence from a school-wide comparative study. BMC Med Educ 21 , 495 (2021). https://doi.org/10.1186/s12909-021-02909-z

Download citation

Received : 31 March 2021

Accepted : 26 August 2021

Published : 16 September 2021

DOI : https://doi.org/10.1186/s12909-021-02909-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Dental education
• Online learning
• COVID-19 pandemic
• Instructional strategies
• Interaction
• Learning performance

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

A Quantitative Study on the Impact of Online Learning on Reading Comprehension Skills

• Open Access
• First Online: 11 May 2023

Cite this chapter

You have full access to this open access chapter

• Ranya Ahmed El Haddad 12 &
• Sa’Ed Mohammad Issa Salhieh 12  

Part of the book series: Lecture Notes in Civil Engineering ((LNCE,volume 320))

7878 Accesses

1 Citations

This quantitative study aims to investigate the relationship between e-education and reading comprehension skills acquisition. It also examines if the previous relationship may impact students’ results in the exams. It also analyses the relationship between students’ knowledge in ICT and their perception and acceptance of online education. To collect data, A survey was sent to students to measure their perception of and satisfaction with online learning. Moreover, the marks of 105 students in an on-campus test were compared to the marks of another one they did online during the pandemic. The study agreed with the previous studies that e-learning can impact the reading skills positively and that students are getting aware of its educational benefits. On the other hand, the study did not agree with other studies about students’ knowledge of ICT and how it can positively impact their perception of online education. The study showed that although secondary students have sufficient knowledge of ICT, they do not have positive perceptions of online education.

Purpose - to investigate the relationship between e-education and acquiring reading comprehension skills, and if this may impact students’ results in the exams.

Methodology - A quantitative study in which a survey and the scores of two reading exams are analysed.

Findings - this study agreed with other studies about the positive impact of e-learning with some differences regarding students’ satisfaction with IT.

Implications - teachers can integrate interactive websites within instruction and using online games and activities can make students more attentive and less distracted.

Originality/value - although most of the studies have proved that there is a positive relationship between the quality of ICT services and students’ satisfaction with online education, this study disagrees as unlike most of the studies, the study in hand was conducted in a secondary school, not in a university.

You have full access to this open access chapter,  Download chapter PDF

Similar content being viewed by others

A Development of Web-Based Instruction Using Self-directed Learning to Enhance English Reading Comprehension for Undergraduate Students

Reading performance and self-regulated learning of hong kong students: what we learnt from pisa 2009.

What Factors Contribute to Effective Online Higher Education? A Meta-Review

• Reading comprehension
• Reading skills
• Online reading

1 Introduction

During Covid19, most of the world has switched to distance education in fear of more spread of the pandemic. This sudden change has put most of school students in confusion which had an impact on their schooling attitude, hence their exam results. In the UAE, the government has tried their best to eliminate, or at least, lessen this fear by providing teachers with trainings required to overcome these non-precedential circumstances. They also provided schools with facilities and equipment to ensure students’ easy accessibility of resources and materials needed to continue learning as smoothly as possible (Ati & Guessoum, 2010 ).

Reading comprehension is a complex skill taught online as a part of the English course delivered to secondary students. It requires connecting points to create a meaning or meanings that are partially derived from prior knowledge. It is an everyday skill that people practice all the time intentionally or unintentionally, yet at school, students should master reading comprehension skills that are developed in classrooms to understand all subjects and pass their exams (Destari, 2010 ).

1.1 Research Questions

Does on-line education have a significant relationship with students’ levels in reading comprehension?

Is there a significant difference between the scores of reading exams (on-campus and online)?

Is there a relationship between the total scores of the two exams and the reading skills mastered in each learning situation?

Do students’ level of Knowledge in ICT and the Benefits of online education have an impact on their Students’ Rating of Online Education?

2 Literature Review

2.1 conceptual framework.

Many concepts can be discussed in this section to give a comprehensive account of this topic such as: reading comprehension and online education. Reading Comprehension is the capability to read, process, and comprehend written material (Butterfuss et al., 2020 ). Online Education is the use of information technologies and communications to assist in the development and acquisition of knowledge from faraway areas (Basilaia & Kvavadze, 2020 ).

2.2 Theoretical Framework

Many theories have discussed reading comprehension, online learning and students’ perception and attitudes. However, the study in hand will discuss the Structural Theory , The Digital Native Theory , and Behaviourism.

The Structural Theory:

It is hard for L2 students to understand written texts for many reasons such as the limited vocabulary knowledge and the text structure and cohesion. Moreover, the text features can influence cognitive process that govern reading comprehension (Jake Follmer & Sperling, 2018 ) as shown in Fig. 1 .

The structural theory and reading comprehension

The Digital Native Theory:

It is believed that nowadays students are digital natives as they were born during the digital revolution (Von der Heiden et al., 2011 ), so they prefer working and gaming online.

Behaviourism:

A behaviour can be due to some external and/or internal causes (Whiteley, 1961 ). In this case, the external cause is the sudden shift to online education. On the other hand, Skinner ( 2011 ) identified Behaviourism as the philosophy of human behaviour. According to him, behaviour is not about cause-and-effect connection, but it is about a set of functional actions that take place in a certain order such as the pandemic, social distancing and e-learning.

2.3 Literature Review

The Benefits of Online Learning on Reading Comprehension: Recently, using technology has been proved successful in improving students’ levels of performance in many subjects including reading comprehension. Many studies have been conducted to identify the type of impact of online education on reading comprehension. The studies of Zidat and Djoudi ( 2010 ) and Ciampa ( 2012 ), have proved that using technology, multimedia and games increased students’ opportunities to gain more reading skills. Other studies have revealed the important role of online reading in improving the level of performance of poor readers which, consequently, improved their reading comprehension skills.

The Impact of Students’ Satisfaction on Their Levels of Performance: Many studies have confirmed the positive relationship between students’ satisfaction and behaviour, and their levels of performance in different contexts. In their studies, Sapri et al. ( 2009 ), Dhaqane and Afrah ( 2016 ) proved that teaching and learning methods used in the higher education institutes had a significant impact on students’ satisfaction which consequently improved their levels of performance. Another study. Furthermore, the study conducted on Vietnamese College students, Salehi et al. ( 2014 ) found out that students with ICT knowledge can feel comfortable learning online.

3 Methodology

This quantitative study will examine the impact of online education on students’ reading comprehension skills and the impact of their ICT knowledge on their satisfaction and behaviour towards online learning. To do so, the study will compare 10 th graders’ results in reading comprehension prior and during distance learning, and analyse the data collected via a survey that will be dispatched to the same students.

3.1 Quantitative Research Paradigms

Paradigms can be considered the ‘worldview’ or ‘sets of beliefs’ that govern the research approaches and methods and lead to answer the research questions (Cohen et al., 2018 ). It is suitable to discuss as it underpins the quantitative approach.

post-Positivism: This theory underpins the quantitative approach as it is concerned with numbers and statistics. According to Alakwe ( 2017 ), post-positivists believe that knowledge is extracted from data that is statistically analysed. This knowledge can be generalizable in similar contexts if showing the same reality observed. This theory is also concerned with decreasing human bias by testing pure data that is not yet interpreted by people.

3.2 Research Methods

There are two instruments used in this study: the first one is 105 10 th graders’ scores in 2 reading comprehension quizzes. The first one was administered at school before the pandemic and the second one was administered online during the pandemic to determine the significance in difference of means using descriptive data and ‘Paired t-test’ on SPSS.

The second tool was a survey to collect data from the same students regarding their attitudes toward the online education phenomenon, the challenges they might have faced while implementing the online education and the level of satisfaction. The survey was conducted anonymously to guarantee objectivity and privacy. The survey used Likert scale in all questions for easier collection of responses.

The survey was adopted from two published studies Footnote 1 : (Simpson, 2012 ; Al-Azawei & Lundqvist, 2015 ). Surveys are used to collect data in the quantitative approach due to the vast development in technology (Mathers et al., 2009 ).

3.3 Sampling

A sample is a part of the population chosen to represent the whole population. The population targeted is 10 th graders, and the sample is 105 female students in a private school in Ajman. There are many types of sampling, but the researcher used the convenience sampling technique due to the nature and logistics of the study during the pandemic (Acharya et al., 2013 ).

4 Result Analysis and Discussion

The study showed that there is a significant positive relationship between online education and students’ improvement in reading skill, yet their satisfaction with and perception of online education is not necessarily congruent with the ICT services provided.

4.1 Question No. 1 and Sub Questions A&B

To answer Q.1, sub-questions A&B will be answered first to be able to find out if there is an impact of online teaching on students’ levels of performance in reading comprehension skills.

Sub-Question A: Is there a significant difference between the scores of reading exams (on-campus and online)? The null hypotheses are: H0: “there is no significant difference in mean between the scores of on-campus reading test and the online reading test” while the alternative hypothesis (H1) is: “there is a significant difference in mean between the scores of on-campus reading test and the online reading test’. A ‘ paired t-test ’ was conducted to confirm or reject the null hypothesis ( H0 ) (Table 1 ).

As the significance factor is P =.732 is higher than α = .05 (P > α), it means that there is no statistically significant difference in means of the scores of the two tests, so the previous results failed statistically to reject the null hypothesis which states that “there is no significant difference in mean between the scores of on-campus’ reading test and the online reading test” with 95% confidence.

Sub Question B: Is there a relationship between the total scores of the two exams and the reading skills mastered in each learning context? A correlation test will be used to answer the question.

To determine the relationship between the previous variables, correlation tests will be used. The null hypothesis (H0) is “there is no significant relationship between reading skills acquired in each educational context and the tests conducted”. P = 0, while the alternative hypothesis (H1) is: “there is a significant relationship between reading skills acquired in each educational contexts and the tests conducted” P ≠ 0).

The following Tables 2 and 3 , show an overall statistically significant positive relationship between the acquired reading skills and the scores of reading tests whether on-campus or online. There is also a significant difference in means between the reading skills acquired online and those acquired at school in favor for the online context.

Does on-line education have a significant relationship with students’ levels in reading comprehension? The percentages of students’ attendance will be used as a reflection of the impact of e-learning as students used to join classes every day. The hypotheses of this questions are the null hypothesis (H0) is: “There is no significant relationship between the percentage of students’ attendance and their scores in the online reading test”. (p = 0), and the alternative hypothesis (H1) is: “There is a significant relationship between the percentage of students’ attendance and their scores in the online reading test” (p ≠ 0). A Pearson correlation test was used to confirm or reject the null hypothesis (Table 4 ).

Coefficient (r) is 0.346. This shows a positive relationship, and it cannot be considered a relatively strong relationship as it is not close to +1. The p value is .001 < alpha value .05. This means that the results statistically reject the null hypothesis and confirms the alternative hypothesis (H1): “Statistically, there is a significant relationship between the percentage of students’ attendance and their scores in the online reading test” Consequently, all the previous results of question 1 and the sub questions A&B prove the fact that there is an overall positive significant relationship between online education and reading comprehension skills acquired and the overall online reading tests score. The previous results conform with Zidat and Djoudi ( 2010 ) and Ciampa ( 2012 ) that the online education is beneficial in relation to reading comprehension skills acquisition.

4.2 Do Students’ Level of Knowledge in ICT and the Benefits of Online Education Have an Impact on Students’ Rating of Online Education?

A survey was conducted on n = (105) to measure students’ knowledge in and satisfaction with ICT. The null hypothesis (H0) is: “There is no significant correlation between students’ level of knowledge in ICT and their Recognition of the online education benefits on their overall rating of online education”. The alternative hypothesis (H1) is: “There is a significant correlation between students’ level of knowledge in ICT and their Recognition of the online education benefits on their overall rating of online education.”

A Linear Regression test was conducted to get answers to the previous question (Table 5 ).

The previous table shows that: P value of the predictor ICT is .432 > alpha value .05 which means that the relationship between ICT and students’ satisfaction is not significant, yet the relationship between the Benefits of online education and satisfaction is significant as P =.001 < α = .05, so there is a significant relationship between the benefits of online education and students’ satisfaction which conforms with Whiteley ( 1961 ) that their satisfaction (effect) is a result of the benefits they are aware of (cause), yet there is no significant relationship between ICT knowledge and students’ satisfaction. This agrees with Skinner ( 2011 ) as students’ negative behaviour and perception of online education is not a result of their lack of knowledge, and it can be a philosophy that has emerged due to other emotional and social factors such as the lack of socialization caused by distance learning.

5 Conclusion, Implications, and Research Significance

Conducting the previous tests, some findings can be highlighted, and some conclusions can be made accordingly.

5.1 Conclusion

The tests conducted show that:

There is a significant impact of online learning on improving the reading comprehension tests’ scores.

These results have confirmed that there is a significant impact of reading skills gained in both educational contexts and the reading tests scores in both contexts.

The study also has proven that there is a positive relation between students’ satisfaction with online education and their improvement in reading skills, yet the relationship between students’ perceptions of online education and the ICT services provided to them.

5.2 Implications and Suggestions

Implications: The hypotheses confirmed in this study can indicate that the types of teaching materials can have a great impact on students’ satisfaction and performance. Using versatile activities and different websites can decrease the boredom and monotony that students might feel in actual classrooms.

Suggestions: Using creative reading material can motivate students to study and practice, so it will be much better to use online reading comprehension resources and activities even after going back to school. Moreover, students can have the chance to study in virtual classes and practice e-reading activities even when they are back to school for at least one school class. This will enable students to enjoy reading and practicing using reading comprehension skills more effectively.

5.3 Research Significance

The results of the study agree partially with the previous studies in that domain, yet it does not agree with the results of other studies about the impact of ICT services on students’ satisfaction with online education. As most studies focused on tertiary students. This study can encourage other researchers to further investigate the context of high school students’ satisfaction and its relationship with ICT services which might reveal new dimensions that might enrich research and become new references to other scholars.

“Learner Differences in Perceived Satisfaction of an Online Learning: an Extension to the Technology Acceptance Model in an Arabic Sample” and “Student Perceptions of Quality and Satisfaction in Online Education”.

Acharya, A., Prakash, A., Saxena, P., & Nigam, A. (2013). Sampling: why and how of it? Indian Journal of Medical Specialities, 4 (2), 330–333.

Google Scholar  

Alakwe, K.O. (2017). Positivism and knowledge inquiry: from scientific method to media and communication research. Specialty Journal of Humanities and Cultural Science, 2 (3), 38–46. Accessed May 31, 2021.

Al-Azawei, A. & Lundqvist, K. (2015) Learner differences in perceived satisfaction of an online learning: an extension to the technology acceptance model in an Arabic sample. Electronic Journal of e-Learning, 13 (5), 412–430. Accessed June 6, 2021.

Prensky, M. (2001). Digital natives, digital immigrants part 1. On the Horizon, 9 (5), 1–6.

Article   Google Scholar  

Aliaga, M. & Gunderson, B. (2003). Interactive statistics (2nd ed.). Pearson education, Inc.

Ati, M. & Guessoum, N. (2010). E-learning in the United Arab Emirates. https://www.researchgate.net/publication/233428047_E-Learning_in_the_United_Arab_Emirates . Accessed May 25, 2021.

Boudalia, M. (2018). Effect of early exposure to technology on student satisfaction with online education. Ph.D. in Management and Technology. Walden University.

Butterfuss, R., Kim, J. & Kendeou, P. (2020). Reading comprehension. Oxford Research Encyclopedia of Education. Accessed May 28, 2021.

Cavanaugh, C., Barbour, M. & Clark, T. (2009). Research and practice in K-12 online learning: a review of open access literature. The International Review of Research in Open and Distributed Learning, 10 (1).

Basilaia, G. & Kvavadze, D. (2020). Transition to online education in schools during a SARS-CoV-2 coronavirus (COVID-19) pandemic in Georgia. Pedagogical Research, 5 (4). Accessed June 20, 2022.

Ciampa, K. (2012). ICANREAD. Journal of Research on Technology in Education, 45 (1), 27–59.

Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education . Routledge.

Coiro, J. (2011). Predicting reading comprehension on the internet. Journal of Literacy Research, 43 (4), 352–392.

D’Elia, E. (2005). Using the results of qualitative surveys in quantitative analysis. SSRN Electronic Journal .

Destari, D. (2010). The effectiveness of internet-based material to teach reading comprehension viewed from learning motivation. graduate degree. Sebelas Maret University Surakarta.

Dhaqane, M. & Afrah, N. (2016). Satisfaction of students and academic performance in Benadir university. Journal of Education and Practice, 7 (24), 59–63. Accessed June 5, 2021.

Evans, J., & Mathur, A. (2005). The value of online surveys. Internet Research, 15 (2), 195–219.

Fidalgo, P., Thormann, J., Kulyk, O. & Lencastre, J. (2020). Students’ perceptions on distance education: a multinational study. International Journal of Educational Technology in Higher Education, 17 (1), 1–18. Accessed June 13, 2021.

Jake Follmer, D. & Sperling, R. (2018). Interactions between reader and text: contributions of cognitive processes, strategy use, and text cohesion to comprehension of expository science text. Learning and Individual Differences, 67 , 177–187. Accessed November 24, 2021.

Keegan, D. (1996). Foundations of distance education.

Mathers, N., Fox, N. & Hunn, A. (2009). Surveys and questionnaires. NIHR RDS for the East Midlands/Yorkshire & Humber. Accessed June 14, 2021.

Rezaee, A. & Sharbaf Shoar, N. (2011). Investigating the effect of using multiple sensory modes of glossing vocabulary items in a reading text with multimedia annotations. English Language Teaching, 4 (2), 25. Accessed June 4, 2021.

Ridzuan, A., Yunus, M., Abdullah, M., Bakar, M. & Ramlan, A. (2018). The relationship between students satisfaction and their academic performance among public relations degree students in UiTM Alor Gajah Melaka. International Journal of Academic Research in Business and Social Sciences, 8 (10), 874–882. Accessed June 5, 2021.

Salehi, H., Shojaee, M. & Sattar, S. (2014). Using E-learning and ICT courses in educational environment: a review. English Language Teaching, 8 (1), 63–70. Accessed June 13, 2021.

Sapri, M., Kaka, A. & Finch, E. (2009). Factors that influence student’s level of satisfaction with regards to higher educational facilities services. Malaysian Journal of Real Estate, 4 (1), 34–51. Accessed June 5, 2021.

Simpson, J. (2012). Student perceptions of quality and satisfaction in online education. Doctor of Philosophy. University of Alabama.

Skinner, B. (2011). About behaviorism. New York: Random House

Sobh, R., & Perry, C. (2006). Research design and data analysis in realism research. European Journal of marketing, 40 (11/12), 1194–1209.

Sterling, K. (2015). Student satisfaction with online learning. Ph.D. in Philosophy. University Of California, Santa Barbara.

Surahman, E. & Sulthoni. (2020). Student satisfaction toward quality of online learning in Indonesian higher education during the Covid-19 pandemic. In 6th international conference on education and technology . Accessed May 28, 2021. https://ieeexplore.ieee.org/abstract/document/9276630

van den Broek, P., Rapp, D., & Kendeou, P. (2005). Integrating memory-based and constructionist processes in accounts of reading comprehension. Discourse Processes, 39 (2–3), 299–316.

Von der Heiden, B., Fleischer, S., Richert, A., & Jeschke, S. (2011). Theory of digital natives in the light of current and future E-learning concepts. International Journal of Emerging Technologies in Learning (IJET), 6 (2), 37.

Whiteley, C. (1961). II—behaviourism. Mind, LXX (278), 164–174.

Zidat, S., & Djoudi, M. (2010). Effects of an online learning on EFL university students’ English reading comprehension. International Review on Computers and Software, 5 (2), 186–192.

Download references

Author information

Authors and affiliations.

The British University in Dubai, Dubai, UAE

Ranya Ahmed El Haddad & Sa’Ed Mohammad Issa Salhieh

You can also search for this author in PubMed   Google Scholar

Corresponding authors

Correspondence to Ranya Ahmed El Haddad or Sa’Ed Mohammad Issa Salhieh .

Editor information

Editors and affiliations.

The British University in Dubai, Dubai, United Arab Emirates

Khalid Al Marri

Farzana Mir

Solomon David

Aos Aljuboori

Rights and permissions

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Reprints and permissions

Copyright information

© 2023 The Author(s)

About this chapter

El Haddad, R.A., Salhieh, S.M.I. (2023). A Quantitative Study on the Impact of Online Learning on Reading Comprehension Skills. In: Al Marri, K., Mir, F., David, S., Aljuboori, A. (eds) BUiD Doctoral Research Conference 2022. Lecture Notes in Civil Engineering, vol 320. Springer, Cham. https://doi.org/10.1007/978-3-031-27462-6_13

Download citation

DOI : https://doi.org/10.1007/978-3-031-27462-6_13

Published : 11 May 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-27461-9

Online ISBN : 978-3-031-27462-6

eBook Packages : Engineering Engineering (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Our approach to learning
• Parent feedback
• Our social purpose
• Creativity and culture
• Expeditions
• Find a school
• Our schools in Europe
• Our schools in the Americas
• Our schools in Asia
• Our schools in the Middle East
• Admissions process
• Why boarding
• Our boarding schools
• Enquire about boarding
• Boarding life
• Boarding admissions and fees
• Discover more
• Latest News
• Explore INSIGHTS today
• A Global Family of Schools
• What is the IB?
• What is the IGCSE?

The Importance of Online Learning to Students and Teachers

As technology advances and user experience improves, so too does the popularity of online education. It’s been proven to be a successful method of learning and offers a number of different benefits when compared with traditional education. While it is beneficial, it also has certain challenges. Understanding the advantages of online learning is important because it could help you make big decisions regarding your or your child’s education. In this article, we will discuss the importance of online learning to students and teachers.

Connects students and teachers internationally

Just like Nord Anglia Education’s Global Campus , an advantage of online learning is it can connect students and teachers around the world. With traditional education, location dictates which classes you could sign up for. This isn’t the case with online learning. You can sign up for classes all over the world, providing a more in-depth understanding of the global industry and helping children build their network and develop an international mindset. Gaining different perspectives and learning about different cultures also aids children’s thinking skills. This highlights the importance of online learning for students because it can open the door to new opportunities and help them develop skills that will serve them well in their future careers.

Offers flexible learning hours

When considering the importance of online learning to students and teachers, flexible learning hours are a big influence. Circumstances mean you can’t always follow the traditional classroom education timetable and being able to learn with more flexible hours can be an important benefit. Online learning offers students the opportunity to learn whenever best suits them. This helps them, and their parents, balance their education with their home life. Transport to school and their parents working hours are two examples of external influences that could impact a traditional education, however, with online learning these don’t have to be a problem. Whether it’s 6 AM or 6 PM, giving children the flexibility to learn when they feel motivated may also increase their ability to retain information.

Considers individual learning patterns

Every student learns differently, and in traditional education, children must adapt to the pace of the class or be left behind. One advantage of online learning is that children have more freedom to work at their own pace, which improves their learning experience and helps them build a better understanding with their teacher. This is also important for the teacher because it helps them structure their classes to suit the individual learning requirements of each child. The outcome: improved grades and a more enjoyable experience for the student and the teacher.

Allows children to create their own learning environment

Children learn better and feel more comfortable learning in an environment of their choosing. This isn’t always possible in a classroom but is one of the key advantages of online learning. As a student or a parent, you understand where you work best, whether it’s in the library, at home or anywhere else. Being able to take a laptop or tablet into your ideal working environment helps children maximise their potential and gain the most from their education. Everyone works differently and some students may prefer the classroom, but for those that don’t, this flexibility can have a positive impact on how they absorb information and help them improve their grades.

Provides students with more control over their learning

Traditional learning methods allow the teacher to decide how they will convey information to their students, but online learning provides more flexibility and gives children control over their education. We all learn differently – some of us prefer quiet study, others like interactive tasks and being challenged under pressure. While there are tasks online, one of the advantages is that it gives students time to teach themselves concepts differently. Some students prefer to learn slowly and explore different learning methods, helping them fully retain information. Once they feel prepared, they can then take online tests or ask their parents to challenge them on what they have learnt.

This article should help you understand the importance of online learning to students and teachers, and whether an online learning experience would be beneficial for your child. If you’re considering a virtual education, you can learn more about the experience we provide through Nord Anglia schools on our virtual schools page.

Want more of the latest insights into education? Read our INSIGHTS publication here !

Related news

The evolution of exams

Happy International Women’s Day!

Dr Leslie Williams, Senior Programme Lead, Social Impact celebrates International Women’s Day 2024.

How to make friends

Want to learn the do’s and don’ts for helping your child make friends? Read our tips and tricks from this quick guide.

Nord Anglia Education launches international metacognition research with Boston College

Want to hear from us?

By joining our mailing list, we can keep you up to date with any future newsletters, events and announcements from our family of 80+ premium schools.

[email protected]

(+44) 020 7131 0000

• About Nord Anglia Education
• Media contacts
• Global Campus
• Press Releases

We use cookies to improve your online experiences. To learn more and choose your cookies options, please refer to our cookie policy .

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Future of undergraduate education for sustainable development goals: impact of perceived flexibility and attitudes on self-regulated online learning.

Share and Cite

Demir, K. Future of Undergraduate Education for Sustainable Development Goals: Impact of Perceived Flexibility and Attitudes on Self-Regulated Online Learning. Sustainability 2024 , 16 , 6444. https://doi.org/10.3390/su16156444

Demir K. Future of Undergraduate Education for Sustainable Development Goals: Impact of Perceived Flexibility and Attitudes on Self-Regulated Online Learning. Sustainability . 2024; 16(15):6444. https://doi.org/10.3390/su16156444

Demir, Kadir. 2024. "Future of Undergraduate Education for Sustainable Development Goals: Impact of Perceived Flexibility and Attitudes on Self-Regulated Online Learning" Sustainability 16, no. 15: 6444. https://doi.org/10.3390/su16156444

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Exploring an online metacognitive intervention on young and mature employees: A preliminary study

• Betsy Ng + −
• Kimberly Hannah Siacor + −
• Diwi Abbas + −
• Cheryl Yeoh + −
• Dion Goh + −

With sufficient metacognitive knowledge and action, employees are better equipped in their time and work management, avoiding any unproductive behaviour. Metacognitive strategies thus equip them with the need to learn and develop their career resilience, especially the young and mature working adults. As there are limited intervention studies exploring the metacognition of both young and mature employees, the present study investigated a metacognitive intervention to promote adaptability and career resilience. It explored the metacognitive strategies that include planning, monitoring and evaluating and examined the differences of various outcome variables (e.g., adaptability, critical thinking and career resilience). A total of 71 participants were recruited, and they underwent an eight-week metacognitive intervention. Pre- and post-questionnaires were administered before and after the intervention. Key findings indicated that the group difference was insignificant, suggesting the importance of metacognitive strategies in supporting adaptability and career resilience for young and mature employees. Finally, limitations to the study were presented, and recommendations were made for future workplace research and practice.

This paper is in the following e-collection/theme issue:

Published on 29.7.2024 in Vol 8 (2024)

Use of Machine Learning Models to Differentiate Neurodevelopment Conditions Through Digitally Collected Data: Cross-Sectional Questionnaire Study

Authors of this article:

Original Paper

• Silvia Grazioli 1, 2, 3 , MSc   ; 
• Alessandro Crippa 1 , MSc, PhD   ; 
• Noemi Buo 1 , MSc   ; 
• Silvia Busti Ceccarelli 1 , MSc   ; 
• Massimo Molteni 1 , MD   ; 
• Maria Nobile 1 , MD, PhD   ; 
• Antonio Salandi 1 , MD   ; 
• Sara Trabattoni 1 , MD   ; 
• Gabriele Caselli 2, 3 , MSc, PhD   ; 
• Paola Colombo 1 , MSc, PhD  

1 Child Psychopathology Unit, Scientific Institute IRCCS Eugenio Medea, Bosisio Parini, Italy

2 Department of Psychology, Sigmund Freud University, Milan, Italy

3 Studi Cognitivi, Cognitive Psychotherapy School and Research Centre, Milan, Italy

Corresponding Author:

Alessandro Crippa, MSc, PhD

Child Psychopathology Unit

Scientific Institute IRCCS Eugenio Medea

Via Don Luigi Monza, 20

Bosisio Parini, 23842

Phone: 39 031877593

Email: [email protected]

Background: Diagnosis of child and adolescent psychopathologies involves a multifaceted approach, integrating clinical observations, behavioral assessments, medical history, cognitive testing, and familial context information. Digital technologies, especially internet-based platforms for administering caregiver-rated questionnaires, are increasingly used in this field, particularly during the screening phase. The ascent of digital platforms for data collection has propelled advanced psychopathology classification methods such as supervised machine learning (ML) into the forefront of both research and clinical environments. This shift, recently called psycho-informatics, has been facilitated by gradually incorporating computational devices into clinical workflows. However, an actual integration between telemedicine and the ML approach has yet to be fulfilled.

Objective: Under these premises, exploring the potential of ML applications for analyzing digitally collected data may have significant implications for supporting the clinical practice of diagnosing early psychopathology. The purpose of this study was, therefore, to exploit ML models for the classification of attention-deficit/hyperactivity disorder (ADHD) and autism spectrum disorder (ASD) using internet-based parent-reported socio-anamnestic data, aiming at obtaining accurate predictive models for new help-seeking families.

Methods: In this retrospective, single-center observational study, socio-anamnestic data were collected from 1688 children and adolescents referred for suspected neurodevelopmental conditions. The data included sociodemographic, clinical, environmental, and developmental factors, collected remotely through the first Italian internet-based screening tool for neurodevelopmental disorders, the Medea Information and Clinical Assessment On-Line (MedicalBIT). Random forest (RF), decision tree, and logistic regression models were developed and evaluated using classification accuracy, sensitivity, specificity, and importance of independent variables.

Results: The RF model demonstrated robust accuracy, achieving 84% (95% CI 82-85; P <.001) for ADHD and 86% (95% CI 84-87; P <.001) for ASD classifications. Sensitivities were also high, with 93% for ADHD and 95% for ASD. In contrast, the DT and LR models exhibited lower accuracy (DT 74%, 95% CI 71-77; P <.001 for ADHD; DT 79%, 95% CI 77-82; P <.001 for ASD; LR 61%, 95% CI 57-64; P <.001 for ADHD; LR 63%, 95% CI 60-67; P <.001 for ASD) and sensitivities (DT: 82% for ADHD and 88% for ASD; LR: 62% for ADHD and 68% for ASD). The independent variables considered for classification differed in importance between the 2 models, reflecting the distinct characteristics of the 3 ML approaches.

Conclusions: This study highlights the potential of ML models, particularly RF, in enhancing the diagnostic process of child and adolescent psychopathology. Altogether, the current findings underscore the significance of leveraging digital platforms and computational techniques in the diagnostic process. While interpretability remains crucial, the developed approach might provide valuable screening tools for clinicians, highlighting the significance of embedding computational techniques in the diagnostic process.

Introduction

Child and adolescent psychopathology entails a complex diagnostic process that includes clinical observations, behavioral assessments, medical history evaluations, cognitive testing, and familial contexts. This comprehensive approach aligns with the recommendations outlined in the National Institute for Health and Care Excellence Guidelines [ 1 ], emphasizing the significance of a multifaceted understanding of these conditions.

In contemporary diagnostic practice, the adoption of digital technologies has gained remarkable prominence, particularly in the digital administration of caregiver-rated questionnaires focused on capturing neurodevelopmental and behavioral symptoms [ 2 , 3 ]. This shift toward digitalization (often called psycho-informatics) is fueled by the gradual incorporation of computational devices into clinical workflows. These tools enable assessments encompassing various aspects, including psychological testing and psychometry [ 4 ]. Furthermore, the onset of the COVID-19 pandemic has further supported the adoption of digital tools for telehealth support, particularly within the mental health sector [ 5 ]. Digitization also includes aspects that cut across diagnoses, such as socio-anamnestic information relevant to neurodevelopmental conditions [ 6 ].

Over the past 2 decades, there has been a significant rise in the application of advanced classification methods, such as supervised machine learning (ML), to enhance diagnostic research in the behavioral sciences [ 1 , 3 , 7 - 16 ]. Most of these studies have applied ML-based models to different types of data (eg, home videos and child or adult diagnostic testing), reaching excellent classification accuracies [ 12 - 17 ]. Supervised ML involves the development of algorithms that acquire knowledge from previous experiences to simulate human cognitive processes. ML techniques have been also used to analyze data collected through digital platforms.

In relation to these advancements, recent work from our group has effectively identified children who received a clinical diagnosis of attention-deficit/hyperactivity disorder (ADHD) with an accuracy of up to 82%, using a simple supervised ML approach with decision trees (DT), based on parent- and teacher-reported child behavior data submitted through an internet-based system [ 3 ]. Ben-Sasson and Yom-Tov [ 18 ] also adopted DT to address a possible early identification of autism spectrum disorder (ASD) through internet-based queries posed by parents suspecting that their children could have that diagnosis; the classifier achieved an area under the curve of 0.82, indicating good predictive accuracy for identifying ASD risk based on parental narratives. Duda et al [ 19 ] recruited through crowdsourcing a large sample of parents of children with only ASD or only ADHD to test whether a supervised ML algorithm could differentiate the 2 diagnoses. Using Social Responsiveness Scale items, the linear discriminant analysis and elastic net classifiers achieved an area under the curve of 0.81.

Nevertheless, previous research conducted on adults with ADHD identified accurate classification models based on questionnaire scores. Trognon and Richard [ 10 ] developed a psychometric screening scale for the identification of adult ADHD based on DSM-5 ( Diagnostic and Statistical Manual of Mental Disorders [Fifth Edition]) diagnostic criteria. They tested an XGBoost classifier to obtain a predictive model for subjects with ADHD compared with controls. The questionnaire scores–based classifier reached an accuracy of 0.98, based on a training set of 154 subjects. Finally, Christiansen et al [ 11 ] tested supervised models on the Conners’ Adult ADHD Rating Scale for differentiating between participants with ADHD, obesity, problematic gambling, and a control group. The models reached a predictive accuracy of 0.82 (support vector machine), 0.80 (LightGBM), and 0.79 (logistic regression [LR]).

When testing ML classification models, a relevant focus should be placed on interpretability and accuracy [ 17 ]. The first deals with the need for researchers and clinicians to fully understand the relationship between inputs and outputs, which provides an automated decision-making process performed by an ML model. In this context, a model is interpretable when it provides all the critical information about between-variable relationships gained through the learning process. Conversely, accuracy describes how well the ML model performs in providing correct class membership predictions, and it increases with model complexity (hence, with less interpretability) [ 17 ]. One or the other must be favored when setting a trade-off between classification accuracy and model interpretability [ 17 ]. Traditionally, psychology has primarily focused on explanatory (interpretable) modeling, seeking to understand the causal underpinnings of behavior. However, this emphasis on explanation has often led to models that lack meaningful predictive capacity, raising questions about the robustness and generalizability of psychological research [ 20 ].

Within this framework, traditional statistical models like LR have been widely used for clinical classification purposes (ie, for predicting the probability that an observation belongs to 1 of 2 possible classes) because of their open interpretability [ 17 ]. Nonetheless, the adoption of less transparent ML methods, such as DT and random forests (RF), has garnered significant attention due to their ability to capture more complex patterns within data. DT and RF often outperform LR, especially when dealing with categorical predictors [ 18 ]. Moreover, recent technological advancements enable the enhancement of interpretability in “black box” models using the Shapley additive explanations (SHAP) framework [ 12 ]. The SHAP analysis evaluates the influence of classification features in augmenting the likelihood of accurate predictions, allowing researchers to gain more insight into ranking factors that make a diagnosis probable [ 12 ].

Given these premises, the hypothesis examined in this study is whether an innovative computational psychometrics framework could exploit the potential of the ML approach to digitally collect data to support the clinical assessment of neurodevelopmental conditions such as ADHD and ASD. To this end, we developed classification ML models to identify either children with ADHD or with ASD using parent-reported socio-anamnestic questionnaires collected through the first Italian internet-based comprehensive screening tool for neurodevelopmental disorders and emotional and behavioral problems, the Medea Information and Clinical Assessment On-Line (MedicalBIT) [ 6 ].

Recruitment

In this retrospective, single-center observational study, we considered socio-anamnestic data from a sample of children and adolescents referred for suspected neurodevelopmental conditions at the Scientific Institute “IRCCS Eugenio Medea”—Associazione La Nostra Famiglia in Bosisio Parini (Lecco, Italy)—between October 2018 and May 2022. A workflow of the diagnostic procedure is shown in Figure 1 .

Sample Description

Participants included 1688 children and adolescents (n=591, 35% females) living in Northern Italy, aged 2 to 16 years (mean 8, SD 3 years), and who were referred for suspected neurodevelopmental and psychopathological diagnosis. All participant-related data were obtained remotely by parents using the MedicalBIT platform, except for the attending clinician’s categorical diagnosis input into the platform. At our Institute, participants underwent a complete neuropsychiatric evaluation and received one or more clinical diagnoses following the DSM-5 criteria [ 19 ]. In some cases, symptom presentation fell below the diagnostic threshold, and the children did not receive a categorical diagnosis. Possible diagnoses comprised ADHD, ASD, anxiety disorders, communication disorders, developmental co-ordination disorders, emotional and behavioral disorders, intellectual disability, mood disorders, obsessive-compulsive disorder, sleep disorders, and specific learning disorders. Each diagnostic label could represent a single condition or a comorbid presentation ( Figure 2 ). The prevalence of ADHD and ASD diagnoses, regardless of single or comorbid presentations, was equal, accounting for 16% (n=270/1688) of the sample each.

Independent Variables

The independent variables were derived from a socio-anamnestic questionnaire tapping clinical, biological, environmental, and developmental information. These variables were selected from standard clinical practice and were completed by the participants’ parents before accessing the Institute using the MedicalBIT internet-based platform [ 6 ]. It is important to note that the questionnaire items were coded such that higher values corresponded to higher risk levels. Consequently, the item values were aggregated to obtain variables that quantified risk levels in positive familiarity, pre- and perinatal risk, developmental concerns, early health problems, and stressful life events ( Table 1 ).

a APGAR: appearance, pulse, grimace, activity, and respiration.

b ADHD: attention-deficit/hyperactivity disorder.

c ASD: autism spectrum disorder.

Outcome Variables

Clinicians’ diagnostic conclusion regarding the presence or absence of ADHD or ASD was considered as a dichotomous outcome in separate classification models, that are (1) the “ADHD” group comprised participants with a clinical diagnosis of ADHD and possible comorbid conditions; the “non-ADHD” group comprised participants without a clinical diagnosis of ADHD, that is, subjects who received other neuropsychiatric diagnoses or no categorical diagnosis, and (2) the “ASD” group comprised participants with a clinical diagnosis of ASD and possible comorbid conditions; the “non-ASD” group comprised participants without a clinical diagnosis of ASD, that is, subjects who received other neuropsychiatric diagnoses or no categorical diagnosis.

Statistical Analysis

Preliminary data handling.

Data handling and statistical analyses were done through R software (version 4.1.2; R Core Team) [ 21 ]. Missing data were imputed using the 10 nearest neighbors averaging [ 22 ].

Classification Models

Separate classification models were obtained, addressing the clinical and research questions, that are (1) “should a new help-seeking child be diagnosed with ADHD, considering the parent-reported socio-anamnestic information?” and (2) “should a new help-seeking child be diagnosed with ASD, considering the parent-reported socio-anamnestic information?”

Random Forest Models

RF models were performed through the “randomForest” toolbox [ 23 ], as previously done [ 3 ]. RF is an ensemble learning technique that generates many DTs and aggregates the results. To prevent overfitting, 2 layers of randomness are added in the procedure through bagging: (1) a bootstrap sample of the data set is considered in each tree (the data that are not considered in the bootstrap sample are called out of bag [OOB]); (2) a subset of mtry-independent variables are selected at each tree node. New data categories are predicted by aggregating all predictions performed by the trees, that is, choosing the majority of the voted categories [ 23 ]. In the tuning phase of the model selection, a leave-one-out cross-validation (LOOCV) approach was applied [ 24 ]. Finally, a SHAP analysis was performed to gain insights into the interpretability of the model [ 25 ]. SHAP values are computed by comparing the model’s predictions with and without a particular feature, and this process is repeated iteratively for each feature and sample in the data set. The magnitude of these values reflects the strength of the effect [ 25 ].

Decision Trees

After conducting RF analyses, DT models were computed. The DT, characterized by a flowchart-like structure, is constructed by considering the entire data set positioned at the top of a “root” node. At each decision point, observations meeting the specified splitting condition are allocated to the left branch, while those not meeting the condition are directed to the right branch [ 26 ]. Information gain is a node impurity measure for selecting attributes and dividing each node, continuing until the terminal node, referred to as the “leaf,” is reached [ 26 ]. Finally, the algorithm assigns the most frequently observed class in each leaf as the classification prediction [ 26 ].

Logistic Regression Models

LR models were used in addition to the DT and RF models. LR is a traditional statistical method widely used for binary classification tasks. It models the probability of a binary outcome (presence or absence of the considered diagnosis) based on one or more predictor variables. In our study, LR was applied using the “glm” function in R.

Testing the Classification Accuracy

Fixed training and test set.

To test the classification accuracy of the previously described models, we used 70% of the whole data set as a training set and the remaining 30% as a test set—the 2 subsamples did not present overlapping subjects. The classification performances of the selected models were evaluated considering the following information on the test set:

• NIR: the no information rate (NIR) represents the largest proportion of the observed classes, indicating the accuracy achievable by always predicting the majority class label.
• The P value of Accuracy>NIR: a hypothesis test result to evaluate whether the classification accuracy performed by the algorithm is greater than the rate of the largest class (NIR).

Five-Fold Cross-Validation

An additional cross-validation step was performed to test the results’ robustness. The whole data set was randomly split into 5 folds, and the 3 classification models were performed on each independent fold. The classification performances were calculated on each test set, and the mean performance values were estimated.

Ethical Considerations

The study was approved by the Institute’s Ethical Review Board (protocol number 7/23, “Comitato Etico IRCCS E. Medea—Sezione Scientifica Associazione La Nostra Famiglia”). The research was conducted following guidelines and regulations depicted in the Declaration of Helsinki. The study data are deidentified, and no identification of individual participants in any images of the paper is possible. All the participant’s parents or legal guardians were informed of the aim of the study. Each subject was free to participate voluntarily and gave their written informed consent to the minor’s participation. No monetary compensation was provided for participating in the study.

The maximum percentage of missing data per subject was 36% (3 subjects). Table 2 depicts the sample’s demographic characteristics, considering the total sample and stratification by ADHD and ASD diagnosis.

a ADHD: attention-deficit/hyperactivity disorder.

b ASD: autism spectrum disorders.

Random Forest

Table 3 shows the RF classification models’ performances. Figure 3 shows the SHAP values (ie, the most important independent variables identified by the RF in accurately classifying the diagnoses).

a SHAP: Shapley additive explanations.

c NIR: no information rate.

d ASD: autism spectrum disorders.

Decision Tree

Table 4 shows the DT model results and performances on the test sets.

b NIR: no information rate.

c ASD: autism spectrum disorders.

Logistic Regression

Table 5 shows the LR model results and performances.

a OR: odds ratio.

Principal Findings

The primary objective of our study was to develop accurate classification models for the diagnosis of ADHD and ASD within a sample referred for clinical evaluation. To this end, we used an ML approach to analyze internet-based parent-reported socio-anamnestic questionnaires.

Our ML models reached overall reasonable classifications in the test sets for both ADHD and ASD. The RF models exhibited classification accuracies of 84% for ADHD and 86% for ASD, respectively, with high sensitivities (93% for ADHD and 95% for ASD). On the other hand, the DT and LR models reached lower accuracy rates, with 74% and 61% accuracy for ADHD and 79% and 63% for ASD, respectively. The DT and LR models also demonstrated lower sensitivities (82% and 62% for ADHD and 88% and 68% for ASD).

In the 5-fold experiment, all models showed a decline in predictive accuracy, as could be expected due to smaller sample sizes. Nevertheless, the RF model continued to exhibit greater accuracy than other models. Concerning the different levels of accuracy reached by our 3 ML models, it is crucial to acknowledge both the advantages and disadvantages of RF, DT, and LR. One of the distinctive features of RF models is that they can effectively capture complex relationships within the data that may elude human interpretation [ 17 ]. For this reason, RF models can occasionally be considered difficult to interpret. This characteristic needs adequate consideration in the clinical context because the primary aim is to provide clinicians with an accurate “first glance” tool that supports them in forming initial diagnostic impressions.

Notwithstanding their eventual interpretability, RF models are remarkably effective in distinguishing different classes, thus representing an asset in psychopathology diagnosis. Conversely, as mentioned above, the DT and LR models are also readily interpretable for clinicians less familiar with ML techniques [ 17 ]. Therefore, the choice of approach depends on the decisional context and the desired degree of interpretability. In this study, we preferred greater levels of classification accuracy over the readiness of the classification process. However, a noteworthy option to mitigate the interpretability concern associated with RF models is provided by SHAP analysis. By assigning an important value to each feature in the classification model, SHAP analysis directly compares RF and other models regarding their interpretability.

Although slightly different in the achieved performance, the 3 models identified sex as the strongest predictor for both ADHD (all 3 models) and ASD (DT and LR models). It is well documented that males are more likely to be diagnosed with both ADHD [ 27 ] and ASD [ 28 ] than females. Interestingly, SHAP analysis indicated a relatively consistent ranking of features for RF models across the 2 clinical diagnoses. After sex, which showed by far the highest discriminative ability among the cases, the presence of pre- and perinatal risk and other developmental concerns featured as influential predictors of both ADHD and ASD classes. Not surprisingly, given the significant heritability of the 2 conditions, having a family member with reported difficulties was also a relevant predictor of the classification.

On the other hand, DT and LR models identified feature rankings that were, except for sex, significantly different for ADHD and ASD classification. This discrepancy could be due to the underlying assumptions of the different ML methodologies. Whereas LR models assume linear relationships between predictors and outcomes, DT and RF models could exploit nonlinear relationships and interactions within the data [ 18 ]. Consequently, some degree of variation in predictor ranking is expected, further highlighting the diverse nature of insights gained from different analytical methodologies. Finally, it should be remembered that it is impossible to conclude the causality and direction of the interrelations among predictors in the ML model.

Comparison With Previous Work

Our RF model’s accuracy was in line with previous ML classification approaches to questionnaire data [ 10 , 11 , 15 , 16 , 29 ] and other data sources [ 30 - 37 ].

Nevertheless, these classification models outperformed recent work from our group, where we identified children with ADHD with an accuracy of up to 82% using a DT-based supervised ML approach [ 3 ]. Despite some methodological differences, the higher level of accuracy obtained in the current work underscores the potential of RF models in increasing the precision of computer-aided diagnosis. Altogether, this pattern of findings suggests that the RF model outperformed both the DT and LR models in effectively categorizing neurodevelopmental conditions based on parent-reported socio-anamnestic information, as highlighted by previous studies [ 27 , 28 , 38 ].

Implications for Clinical Practice and Future Research

In the domain of child and adolescent neuropsychiatry, the diagnostic process includes an initial stage where anamnestic, sociodemographic, and behavioral data need to be collected. This data gathering can be remotely performed through internet-based parent reports, as evidenced by previous studies [ 5 , 6 ]. With this regard, the MedicalBIT platform currently represents the first Italian internet-based screening instrument for child and adolescent neuropsychiatric conditions [ 6 ]. As the data are compiled in databases within MedicalBIT, the exploitation of ML models can prompt the classification of the probable diagnostic risk associated with new subjects seeking assistance. The significant predictive value of the models developed in this study might be valuable to support the clinical practice of diagnosing neurodevelopmental conditions.

Limitations

Despite the encouraging findings, this study is not free of limitations. First, our ML models exclusively rely on parent-reported data. Existing literature [ 37 ] has previously indicated that the reliability of these reports could be negatively influenced by factors such as the possibility of accessing digital tools, intrinsic comprehension difficulties, or general parental educational attainment. Second, our sample exclusively included children and adolescents from a geographically restricted region (Northern Italy). The generalizability of current findings to populations from different areas needs cautious consideration. Third, the relatively low occurrence of ADHD-ASD comorbidity in our cohort prevents us from developing classification models tailored for more nuanced diagnostic presentations, such as either ADHD- or ASD-only versus ADHD-ASD comorbid presentation. Therefore, future extensions of this study should consider including broader cohorts of participants to consider this possibility.

Conclusions and Future Advancements

Within the rapidly evolving context of “psycho-informatics,” we believe that the current work represents a noteworthy effort in the realm of computational psychometrics [ 28 ]. Through an exploration of remotely collected parent-reported socio-anamnestic data, the current research has revealed promising avenues for enhancing the diagnostic process of neurodevelopmental and psychopathological conditions. Integrating digital platforms for data collection and ML could offer clinicians a dynamic tool supporting their diagnostic decisions. Within the health care systems, clinical teams confront a scarcity of personnel, with high emotional and cognitive demands for the actual staff [ 38 ]. In this context, this research represents a preliminary effort to mitigate the clinicians’ workload by automating specific tasks (such as data collection and analysis). If proven effective, this approach could leave more time for clinicians to nurture the essential patient-clinician bond, a facet that remains irreplaceable by artificial intelligence technologies.

Acknowledgments

SG performed statistical analyses and wrote the first draft. SBC, NB, MN, AS, ST, MM, and PC contributed clinical knowledge regarding neurodevelopmental and psychopathological conditions and telemedicine. AC, GC, and PC contributed scientific knowledge on health care technology. SBC, NB, and PC contributed to the development of the telemedicine platform through clinical consultation for variable selection. All authors revised and approved the final manuscript.

Data Availability

The data sets analyzed during this study are available from the corresponding author on reasonable request.

Conflicts of Interest

None declared.

• National Collaborating Centre for Mental Health (UK). Attention Deficit Hyperactivity Disorder: Diagnosis and Management of ADHD in Children, Young People and Adults. Leicester, UK. British Psychological Society (UK); 2009.
• Alavi N, Yang M, Stephenson C, Nikjoo N, Malakouti N, Layzell G, et al. Using the online psychotherapy tool to address mental health problems in the context of the COVID-19 pandemic: protocol for an electronically delivered cognitive behavioral therapy program. JMIR Res Protoc. 2020;9(12):e24913. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Grazioli S, Crippa A, Rosi E, Candelieri A, Ceccarelli SB, Mauri M, et al. Exploring telediagnostic procedures in child neuropsychiatry: addressing ADHD diagnosis and autism symptoms through supervised machine learning. Eur Child Adolesc Psychiatry. 2024;33(1):139-149. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Markowetz A, Błaszkiewicz K, Montag C, Switala C, Schlaepfer T. Psycho-informatics: big data shaping modern psychometrics. Med Hypotheses. 2014;82(4):405-411. [ CrossRef ] [ Medline ]
• Sherwood A, MacDonald B. A teleneuropsychology consultation service model for children with neurodevelopmental and acquired disorders residing in rural state regions. Arch Clin Neuropsychol. 2020;35(8):1196-1203. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Colombo P, Busti Ceccarelli S, Pacchiarini S, Cribellati S, Molteni M. A web platform for standardized data acquisition, processing, and export in the child psychopathology clinical routine (MedicalBIT): design and implementation study. JMIR Form Res. 2022;6(7):e36757. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Briganti G, Le Moine O. Artificial intelligence in medicine: today and tomorrow. Front Med (Lausanne). 2020;7:27. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Crippa A, Salvatore C, Perego P, Forti S, Nobile M, Molteni M, et al. Use of machine learning to identify children with autism and their motor abnormalities. J Autism Dev Disord. Jul 2015;45(7):2146-2156. [ CrossRef ] [ Medline ]
• Crippa A, Salvatore C, Molteni E, Mauri M, Salandi A, Trabattoni S, et al. The utility of a computerized algorithm based on a multi-domain profile of measures for the diagnosis of attention deficit/hyperactivity disorder. Front Psychiatry. 2017;8:189. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Trognon A, Richard M. Questionnaire-based computational screening of adult ADHD. BMC Psychiatry. 2022;22(1):401. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Christiansen H, Chavanon M, Hirsch O, Schmidt MH, Meyer C, Müller A, et al. Use of machine learning to classify adult ADHD and other conditions based on the conners' adult ADHD rating scales. Sci Rep. 2020;10(1):18871. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Dziugaite G, Ben-David S, Roy D. Enforcing interpretability and its statistical impacts: trade-offs between accuracy and interpretability. ArXiv. Preprint posted online on October 28, 2020. 2020. [ FREE Full text ]
• Yarkoni T, Westfall J. Choosing prediction over explanation in psychology: lessons from machine learning. Perspect Psychol Sci. 2017;12(6):1100-1122. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Breiman L. Random forests. Mach Learn. 2001;45:5-32. [ FREE Full text ]
• Caly H, Rabiei H, Coste-Mazeau P, Hantz S, Alain S, Eyraud J, et al. Machine learning analysis of pregnancy data enables early identification of a subpopulation of newborns with ASD. Sci Rep. 2021;11(1):6877. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Akter T, Khan M, Ali M, Satu M, Uddin M, Moni M. Improved machine learning based classification model for early autism detection. Dhaka, Bangladesh. IEEE; 2021. Presented at: 2021 2nd International Conference on Robotics, Electrical and Signal Processing Techniques (ICREST); 2021 Jan 5-7:742-747; Dhaka, Bangladesh. [ CrossRef ]
• Young SR, Maddocks DLS, Caemmerer JM. Computer-enhanced practice: The benefits of computer-assisted assessment in applied clinical practice. Prof Psychol Res Pract. 2022;53(4):387-397. [ CrossRef ]
• Ben-Sasson A, Yom-Tov E. Online concerns of parents suspecting autism spectrum disorder in their child: content analysis of signs and automated prediction of risk. J Med Internet Res. 2016;18(11):e300. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Duda M, Haber N, Daniels J, Wall DP. Crowdsourced validation of a machine-learning classification system for autism and ADHD. Transl Psychiatry. 2017;7(5):e1133. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cao X, Cao J. Commentary: machine learning for autism spectrum disorder diagnosis - challenges and opportunities - a commentary on Schulte-Rüther et al. (2022). J Child Psychol Psychiatry. 2023;64(6):966-967. [ CrossRef ] [ Medline ]
• Battle DE. Diagnostic and statistical manual of mental disorders. Codas. 2013;25(2):191-192. [ FREE Full text ] [ CrossRef ] [ Medline ]
• de JE, van DLM. An Introduction to Data Cleaning With R. Hague, Netherlands. Statistics Netherlands; 2013.
• Franzin A, Sambo F, Di Camillo B. bnstruct: an R package for Bayesian network structure learning in the presence of missing data. Bioinformatics. 2017;33(8):1250-1252. [ CrossRef ] [ Medline ]
• Liaw A, Wiener M. Classification and regression by randomForest. R News. 2002;2(3):18-22. [ FREE Full text ]
• Refaeilzadeh P, Tang L, Liu H, Liu L, Özsu M. Cross-validation. Encyclopedia of Database Systems. 2009;5:532-538. [ CrossRef ]
• Perochon S, Di Martino JM, Carpenter KLH, Compton S, Davis N, Eichner B, et al. Early detection of autism using digital behavioral phenotyping. Nat Med. 2023;29(10):2489-2497. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Patel HH, Prajapati P. Study and analysis of decision tree based classification algorithms. IJCSE. 2018;6(10):74-78. [ CrossRef ]
• Devika VG, Chinnaiyan R. Optimized machine learning classification approaches for prediction of autism spectrum disorder. Ann Autism Dev Disord. 2020;1(1):1001. [ FREE Full text ]
• Albahri AS, Hamid RA, Zaidan AA, Albahri OS. Early automated prediction model for the diagnosis and detection of children with autism spectrum disorders based on effective sociodemographic and family characteristic features. Neural Comput & Applic. 2023;35(1):921-947. [ CrossRef ]
• van der Meer D, Hoekstra PJ, van Donkelaar M, Bralten J, Oosterlaan J, Heslenfeld D, et al. Predicting attention-deficit/hyperactivity disorder severity from psychosocial stress and stress-response genes: a random forest regression approach. Transl Psychiatry. 2017;7(6):e1145-e1145. [ CrossRef ]
• Zhang-James Y, Chen Q, Kuja-Halkola R, Lichtenstein P, Larsson H, Faraone SV. Machine-Learning prediction of comorbid substance use disorders in ADHD youth using Swedish registry data. J Child Psychol Psychiatry. 2020;61(12):1370-1379. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Faraone SV, Asherson P, Banaschewski T, Biederman J, Buitelaar JK, Ramos-Quiroga JA, et al. Attention-deficit/hyperactivity disorder. Nat Rev Dis Primers. Aug 06, 2015;1:15020. [ CrossRef ] [ Medline ]
• Elsabbagh M, Divan G, Koh Y, Kim YS, Kauchali S, Marcín C, et al. Global prevalence of autism and other pervasive developmental disorders. Autism Res. 2012;5(3):160-179. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hewitt K, Rodgin S, Loring D, Pritchard A, Jacobson L. Transitioning to telehealth neuropsychology service: considerations across adult and pediatric care settings. Clin Neuropsychol. 2020;34(7-8):1335-1351. [ CrossRef ] [ Medline ]
• Pasca L, Zanaboni M, Grumi S, Totaro M, Ballante E, Varesio C, et al. Impact of COVID-19 pandemic in pediatric patients with epilepsy with neuropsychiatric comorbidities: a telemedicine evaluation. Epilepsy Behav. 2021;115:107519. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pritchard A, Sweeney K, Salorio C, Jacobson L. Pediatric neuropsychological evaluation via telehealth: novel models of care. Clin Neuropsychol. 2020;34(7-8):1367-1379. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tahıllıoğlu A, Bilaç Ö, Uysal T, Ercan E. Who predict ADHD with better diagnostic accuracy?: Parents or teachers? Nord J Psychiatry. 2021;75(3):214-223. [ CrossRef ] [ Medline ]
• O'Connor K, Muller Neff D, Pitman S. Burnout in mental health professionals: a systematic review and meta-analysis of prevalence and determinants. Eur Psychiatry. 2018;53:74-99. [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 15.11.23; peer-reviewed by P Washington, S Young, A Madevska Bogdanova; comments to author 10.01.24; revised version received 27.03.24; accepted 25.04.24; published 29.07.24.

©Silvia Grazioli, Alessandro Crippa, Noemi Buo, Silvia Busti Ceccarelli, Massimo Molteni, Maria Nobile, Antonio Salandi, Sara Trabattoni, Gabriele Caselli, Paola Colombo. Originally published in JMIR Formative Research (https://formative.jmir.org), 29.07.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.