Dataset Information

Massive Open Online Courses (MOOC) Evaluation Methods: Protocol for a Systematic Review.

ABSTRACT:

Background

Massive open online courses (MOOCs) have increased in popularity in recent years. They target a wide variety of learners and use novel teaching approaches, yet often exhibit low completion rates (10%). It is important to evaluate MOOCs to determine their impact and effectiveness, but little is known at this point about the methodologies that should be used for evaluation.

Objective

The purpose of this paper is to provide a protocol for a systematic review on MOOC evaluation methods.

Methods

We will use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P) guidelines for reporting this protocol. We developed a population, intervention, comparator, and outcome (PICO) framework to guide the search strategy, based on the overarching question, "What methods have been used to evaluate MOOCs?" The review will follow six stages: 1) literature search, 2) article selection, 3) data extraction, 4) quality appraisal, 5) data analysis, and 6) data synthesis.

Results

The systematic review is ongoing. We completed the data searches and data abstraction in October and November 2018. We are now analyzing the data and expect to complete the systematic review by March 2019.

Conclusions

This systematic review will provide a useful summary of the methods used for evaluation of MOOCs and the strengths and limitations of each approach. It will also identify gaps in the literature and areas for future work.

International registered report identifier (irrid)

DERR1-10.2196/12087.

SUBMITTER: Foley K

PROVIDER: S-EPMC6427096 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:BACKGROUND:Massive open online courses (MOOCs) have the potential to make a broader educational impact because many learners undertake these courses. Despite their reach, there is a lack of knowledge about which methods are used for evaluating these courses. OBJECTIVE:The aim of this review was to identify current MOOC evaluation methods to inform future study designs. METHODS:We systematically searched the following databases for studies published from January 2008 to October 2018: (1) Scopus, (2) Education Resources Information Center, (3) IEEE (Institute of Electrical and Electronic Engineers) Xplore, (4) PubMed, (5) Web of Science, (6) British Education Index, and (7) Google Scholar search engine. Two reviewers independently screened the abstracts and titles of the studies. Published studies in the English language that evaluated MOOCs were included. The study design of the evaluations, the underlying motivation for the evaluation studies, data collection, and data analysis methods were quantitatively and qualitatively analyzed. The quality of the included studies was appraised using the Cochrane Collaboration Risk of Bias Tool for randomized controlled trials (RCTs) and the National Institutes of Health-National Heart, Lung, and Blood Institute quality assessment tool for cohort observational studies and for before-after (pre-post) studies with no control group. RESULTS:The initial search resulted in 3275 studies, and 33 eligible studies were included in this review. In total, 16 studies used a quantitative study design, 11 used a qualitative design, and 6 used a mixed methods study design. In all, 16 studies evaluated learner characteristics and behavior, and 20 studies evaluated learning outcomes and experiences. A total of 12 studies used 1 data source, 11 used 2 data sources, 7 used 3 data sources, 4 used 2 data sources, and 1 used 5 data sources. Overall, 3 studies used more than 3 data sources in their evaluation. In terms of the data analysis methods, quantitative methods were most prominent with descriptive and inferential statistics, which were the top 2 preferred methods. In all, 26 studies with a cross-sectional design had a low-quality assessment, whereas RCTs and quasi-experimental studies received a high-quality assessment. CONCLUSIONS:The MOOC evaluation data collection and data analysis methods should be determined carefully on the basis of the aim of the evaluation. The MOOC evaluations are subject to bias, which could be reduced using pre-MOOC measures for comparison or by controlling for confounding variables. Future MOOC evaluations should consider using more diverse data sources and data analysis methods. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID):RR2-10.2196/12087.

Project description:IntroductionMassive open online courses (MOOCs) offer a flexible approach to online and distance learning, and are growing in popularity. Several MOOCs are now available, to help learners build on their knowledge in a number of healthcare topics. More research is needed to determine the effectiveness of MOOCs as an online education tool, and explore their long-term impact on learners' professional practice. We present a protocol describing the design of comprehensive, mixed-methods evaluation of a MOOC, 'Q uality I mprovement ( QI ) in H ealthcare : the Case for Change', which aims to improve learner's knowledge and understanding of QI approaches in healthcare, and to increase their confidence in participating, and possibly leading QI projects.Methods and analysisA pre-post study design using quantitative and qualitative methods will be used to evaluate the QI MOOC. Different elements of the RE-AIM (reach, effectiveness and maintenance) and Kirkpatrick (reaction, learning and behaviour) models will be used to guide the evaluation. All learners who register for the course will be invited to participate in the QI MOOC evaluation study. Those who consent will be asked to complete a presurvey to assess baseline QI knowledge (self-report and objective) and perceived confidence in engaging in QI activities. On completion of the course, participants will complete a postsurvey measuring again knowledge and perceived confidence. Feedback on the course content and how it can be improved. A subset of participants will be invited to take part in a follow-up qualitative interview, 3 months after taking the course, to explore in depth how the MOOC impacted their behaviour in practice.Ethics and disseminationThe study has been approved by the University of Bath Human Research Ethics Committee (reference: 2958). Study findings will be published in peer-reviewed journals, and disseminated at conference and departmental presentations, and more widely using social media, microblogging sites and periodicals aimed at healthcare professionals.

Project description:INTRODUCTION:The educational technology of massive open online courses (MOOCs) has been successfully applied in a wide variety of disciplines and are an intense focus of educational research at this time. Educators are now looking to MOOC technology as a means to improve professional medical education, but very little is known about how medical MOOCs compare with traditional content delivery. METHODS:A retrospective analysis of the course evaluations for the Medicine as a Business elective by fourth-year medical students at Southern Illinois University School of Medicine (SIU-SOM) for the 2012-2015 academic years was conducted. This course was delivered by small group flipped classroom discussions for 2012-2014 and delivered via MOOC technology in 2015. Learner ratings were compared between the two course delivery methods using routinely collected course evaluations. RESULTS:Course enrollment has ranged from 6-19 students per year in the 2012-2015 academic years. Student evaluations of the course are favorable in the areas of effective teaching, accurate course objectives, meeting personal learning objectives, recommending the course to other students, and overall when rated on a 5-point Likert scale. The majority of all student ratings (76-95%) of this elective course are for the highest possible choice (Strongly agree or Excellent) for any criteria, regardless if the course was delivered via a traditional or MOOC format. Statistical analysis of these ratings suggests that the Effective Teacher and Overall Evaluations did not statistically differ between the two delivery formats. DISCUSSION:Student ratings of this elective course were highly similar when delivered in a flipped classroom format or by using MOOC technology. The primary advantage of this new course format is flexibility of time and place for learners, allowing them to complete the course objectives when convenient for them. The course evaluations suggest this is a change that is acceptable to the target audience. CONCLUSIONS:This study suggests that learner evaluations of a fourth-year medical school elective course do not significantly differ when delivered by flipped classroom group discussions or via MOOC technology in a very small single center observational study. Further investigation is required to determine if this delivery method is an acceptable and effective means of teaching in the medical school environment.

Project description:IntroductionIncreasing number of Massive Open Online Courses (MOOCs) are being used to train learners at scale in various healthcare-related skills. However, many challenges in course delivery require further understanding, for example, factors exploring the reasons for high MOOC dropout rates, recorded low social interaction between learners and the lack of understanding of the impact of a course facilitators' presence in course engagement. There is a need to generate further evidence to explore these detriments to MOOC course delivery to enable enhanced course learning design. The proposed mixed-methods evaluation of the MOOC was determined based on the MOOC's aims and objectives and the methodological approaches used to evaluate this type of a course. The MOOC evaluation will help appraise the effectiveness of the MOOC in delivering its intended objectives. This protocol aims to describe the design of a study evaluating learners knowledge, skills and attitudes in a MOOCs about data science for healthcare.Methods and analysisStudy participants will be recruited from learners who have registered for the MOOC. On registration, learners will be given an opportunity to opt into the study and complete informed consent. Following completion of the course, study participants will be contacted to complete semistructured interviews. Interviews will be transcribed and coded using thematic analysis, with data analysed using two evaluation models: (1) the reach, effectiveness, adoption, implementation, maintenance framework and the (2) Kirkpatrick model drawing data from pre and post-course surveys and post-MOOC semi-structured interviews. The primary goal of the evaluation is to appraise participants' knowledge, skills and attitude after taking the MOOC.Ethics and disseminationEthics approval for this study was obtained from Imperial College London through the Education Ethics Review Process (EERP) (EERP1617-030). A summary of the research findings will be reported through a peer-reviewed journal and will be presented at an international conference.

Project description:BackgroundSelf-management education is a fundamental aspect in the health care of people with diabetes to develop the necessary skills for the improvement of health outcomes. Patients are required to have the competencies to manage electronic information resources-that is, an appropriate level of digital health literacy. The European project IC-Health aimed to improve digital health literacy among people with diabetes through the cocreation of massive open online courses (MOOCs).ObjectiveWe report the preliminary results obtained in 3 participating countries in the IC-Health project (Italy, Spain, and Sweden) regarding (1) experience of the participants during the cocreation process of MOOCs, (2) perceived changes in their digital health literacy level after using MOOCs, and (3) a preliminary assessment of the acceptability of MOOCs.MethodsThe cocreation of the MOOCs included focus groups with adults and adolescents with diabetes and the creation of independent communities of practice for type 1 diabetes and type 2 diabetes participants aimed to co-design the MOOCs. Quantitative measures of the acceptability of MOOCs, experience in the cocreation process, and increase in digital health literacy (dimensions of finding, understanding, and appraisal) were assessed.ResultsA total of 28 participants with diabetes participated in focus groups. Adults and adolescents agreed that the internet is a secondary source of health-related information. A total of 149 participants comprised the diabetes communities of practice. A total of 9 MOOCs were developed. Acceptability of the MOOCs and the cocreation experience were positively valued. There was a significant improvement in digital health literacy in both adults and adolescents after using MOOCs (P<.001).ConclusionsAlthough the results presented on self-perceived digital health literacy are preliminary and exploratory, this pilot study suggests that IC-Health MOOCs represent a promising tool for the medical care of diabetes, being able to help reduce the limitations associated with low digital health literacy and other communication barriers in the diabetes population.

Project description:BackgroundThe implementation of the National Genomic Medicine Service in the UK has increased patient access to germline genomic testing. Increased testing leads to more genetic diagnoses but does result in the identification of genomic variants of uncertain significance (VUS). The rigorous process of interpreting these variants requires multi-disciplinary, highly trained healthcare professionals (HCPs). To meet this training need, we designed two Massive Open Online Courses (MOOCs) for HCPs involved in germline genomic testing pathways: Fundamental Principles (FP) and Inherited Cancer Susceptibility (ICS).MethodsAn evaluation cohort of HCPs involved in genomic testing were recruited, with additional data also available from anonymous self-registered learners to both MOOCs. Pre- and post-course surveys and in-course quizzes were used to assess learner satisfaction, confidence and knowledge gained in variant interpretation. In addition, granular feedback was collected on the complexity of the MOOCs to iteratively improve the resources.ResultsA cohort of 92 genomics HCPs, including clinical scientists, and non-genomics clinicians (clinicians working in specialties outside of genomics) participated in the evaluation cohort. Between baseline and follow-up, total confidence scores improved by 38% (15.2/40.0) (95% confidence interval [CI] 12.4-18.0) for the FP MOOC and 54% (18.9/34.9) (95%CI 15.5-22.5) for the ICS MOOC (p < 0.0001 for both). Of those who completed the knowledge assessment through six summative variant classification quizzes (V1-6), a mean of 79% of respondents classified the variants such that correct clinical management would be undertaken (FP: V1 (73/90) 81% Likely Pathogenic/Pathogenic [LP/P]; V2 (55/78) 70% VUS; V3 (59/75) 79% LP/P; V4 (62/72) 86% LP/LP. ICS: V5 (66/91) 73% VUS; V6 (76/88) 86% LP/P). A non-statistically significant higher attrition rate was seen amongst the non-genomics workforce when compared to genomics specialists for both courses. More participants from the non-genomics workforce rated the material as "Too Complex" (FP n = 2/7 [29%], ICS n = 1/5 [20%]) when compared to the specialist genomics workforce (FP n = 1/43 [2%], ICS n = 0/35 [0%]).ConclusionsAfter completing one or both MOOCs, self-reported confidence in genomic variant interpretation significantly increased, and most respondents could correctly classify variants such that appropriate clinical management would be instigated. Genomics HCPs reported higher satisfaction with the level of content than the non-genomics clinicians. The MOOCs provided foundational knowledge and improved learner confidence, but should be adapted for different workforces to maximise the benefit for clinicians working in specialties outside of genetics.

Project description:This paper aims to survey language teaching MOOCs that promote intercultural competence (IC). Specifically, the survey aims to identify the keywords most relevant to MOOCs with IC components, the languages taught, the offering countries, and the embedded soft skills. The selection of keywords is important because they can indicate connection between different disciplines. After trialing a broader set of keywords in several rounds of initial search, we identified five keywords that are the most relevant to language education with IC components on MOOCs: culture, intercultural, cross culture, multi culture, and society. Then courses with these five keywords on language learning are selected for further analysis. The results are summarized as follows: (1) Most language teaching MOOCs are found under the keyword “culture,” indicating a strong connection between language education and intercultural communication; (2) In terms of the target languages and the offering countries, it is found that English, Chinese, and Spanish are the major languages widely taught in the context of intercultural competence; China, United States, and Ireland are the three countries which offer the highest numbers of MOOCs in this respect; United States, United Kingdom, and Australia are the three countries which offer the highest numbers of MOOCs of different languages. The results indicate that a limited number of languages and offering countries dominate the language learning MOOCs. The study calls for a plurality of languages and cultures to be taught through MOOCs, making more diversified knowledge systems accessible to global audiences. (3) The language teaching MOOCs not only focus on language but also aim to foster five types of soft skills (language learning skills, communication skills, business and entrepreneurship skills, career development skills, and cultural development skills), suggesting that intercultural competence and its related soft skills are usually important components embedded in such MOOCs courses.