Dataset Information

Scaling up behavioral science interventions in online education.

ABSTRACT: Online education is rapidly expanding in response to rising demand for higher and continuing education, but many online students struggle to achieve their educational goals. Several behavioral science interventions have shown promise in raising student persistence and completion rates in a handful of courses, but evidence of their effectiveness across diverse educational contexts is limited. In this study, we test a set of established interventions over 2.5 y, with one-quarter million students, from nearly every country, across 247 online courses offered by Harvard, the Massachusetts Institute of Technology, and Stanford. We hypothesized that the interventions would produce medium-to-large effects as in prior studies, but this is not supported by our results. Instead, using an iterative scientific process of cyclically preregistering new hypotheses in between waves of data collection, we identified individual, contextual, and temporal conditions under which the interventions benefit students. Self-regulation interventions raised student engagement in the first few weeks but not final completion rates. Value-relevance interventions raised completion rates in developing countries to close the global achievement gap, but only in courses with a global gap. We found minimal evidence that state-of-the-art machine learning methods can forecast the occurrence of a global gap or learn effective individualized intervention policies. Scaling behavioral science interventions across various online learning contexts can reduce their average effectiveness by an order-of-magnitude. However, iterative scientific investigations can uncover what works where for whom.

SUBMITTER: Kizilcec RF

PROVIDER: S-EPMC7334459 | biostudies-literature | 2020 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Scaling up behavioral science interventions in online education.

Kizilcec René F RF Reich Justin J Yeomans Michael M Dann Christoph C Brunskill Emma E Lopez Glenn G Turkay Selen S Williams Joseph Jay JJ Tingley Dustin D

Proceedings of the National Academy of Sciences of the United States of America 20200615 26

Online education is rapidly expanding in response to rising demand for higher and continuing education, but many online students struggle to achieve their educational goals. Several behavioral science interventions have shown promise in raising student persistence and completion rates in a handful of courses, but evidence of their effectiveness across diverse educational contexts is limited. In this study, we test a set of established interventions over 2.5 y, with one-quarter million students, ...[more]

PMID: 32541050

Similar Datasets

Project description:BackgroundMost low and middle income countries (LMICs) are currently not on track to reach the health-related Millennium Development Goals (MDGs). One way to accelerate progress would be through the large-scale implementation of evidence-based health tools and interventions. This study aimed to: (a) explore the barriers that have impeded such scale-up in LMICs, and (b) lay out an "implementation research agenda"--a series of key research questions that need to be addressed in order to help overcome such barriers.MethodsInterviews were conducted with fourteen key informants, all of whom are academic leaders in the field of implementation science, who were purposively selected for their expertise in scaling up in LMICs. Interviews were transcribed by hand and manually coded to look for emerging themes related to the two study aims. Barriers to scaling up, and unanswered research questions, were organized into six categories, representing different components of the scaling up process: attributes of the intervention; attributes of the implementers; scale-up approach; attributes of the adopting community; socio-political, fiscal, and cultural context; and research context.ResultsFactors impeding the success of scale-up that emerged from the key informant interviews, and which are areas for future investigation, include: complexity of the intervention and lack of technical consensus; limited human resource, leadership, management, and health systems capacity; poor application of proven diffusion techniques; lack of engagement of local implementers and of the adopting community; and inadequate integration of research into scale-up efforts.ConclusionsKey steps in expanding the evidence base on implementation in LMICs include studying how to: simplify interventions; train "scale-up leaders" and health workers dedicated to scale-up; reach and engage communities; match the best delivery strategy to the specific health problem and context; and raise the low profile of implementation science.

Project description:BackgroundQuality improvement (QI) evaluations rarely consider how a successful intervention can be sustained long term, nor how to spread or scale to other locations. A survey of authors of randomized trials of diabetes QI interventions included in an ongoing systematic review found that 78% of trials reported improved quality of care, but 40% of these trials were not sustained. This study explores why and how the effective interventions were sustained, spread, or scaled.MethodsA qualitative approach was used, focusing on case examples. Diabetes QI program trial authors were purposefully sampled and recruited for telephone interviews. Authors were eligible if they had completed the author survey, agreed to follow-up, and had a completed a diabetes QI trial they deemed "effective." Snowball sampling was used if the participant identified someone who could provide a different perspective on the same trial. Interviews were transcribed verbatim. Inductive thematic analysis was conducted to identify barriers and facilitators to sustainability, spread, and/or scale of the QI program, using case examples to show trajectories across projects and people.ResultsEleven of 44 eligible trialists participated in an interview. Four reported that the intervention was "sustained" and nine were "spread," however, interviews highlighted that these terms were interpreted differently over time and between participants. Participant stories highlighted the varied trajectories of how projects evolved and how some research careers adapted to increase impact. Three interacting themes, termed the "3C's," helped explain the variation in sustainability, spread, and scale: (i) understanding the concepts of implementation, sustainability, sustainment, spread, and scale; (ii) having the appropriate competencies; and (iii) the need for individual, organizational, and system capacity.ConclusionsChallenges in defining sustainability, spread and scale make it difficult to fully understand impact. However, it is clear that from the beginning of intervention design, trialists need to understand the concepts and have the competency and capacity to plan for feasible and sustainable interventions that have potential to be sustained, spread and/or scaled if found to be effective.

Dataset Information

Scaling up behavioral science interventions in online education.

Publications

Scaling up behavioral science interventions in online education.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets