Dataset Information

Development of an Instructional Design Evaluation Survey for Postgraduate Medical E-Learning: Content Validation Study.

ABSTRACT:

Background

E-Learning has taken a firm place in postgraduate medical education. Whereas 10 years ago it was promising, it now has a definite niche and is clearly here to stay. However, evaluating the effect of postgraduate medical e-learning (PGMeL) and improving upon it can be complicated. While the learning aims of e-learning are evaluated, there are no instruments to evaluate the instructional design of PGMeL. Such an evaluation instrument may be developed by following the Association for Medical Education in Europe (AMEE) 7-step process. The first 5 steps of this process were previously performed by literature reviews, focus group discussion, and an international Delphi study.

Objective

This study will continue with steps 6 and 7 and answer the research question: Is a content-validated PGMeL evaluation survey useful, understandable, and of added value for creators of e-learning?

Methods

There are five phases in this study: creating a survey from 37 items (phase A); testing readability and question interpretation (phase B); adjusting, rewriting, and translating surveys (phase C); gathering completed surveys from three PGMeL modules (phase D); and holding focus group discussions with the e-learning authors (phase E). Phase E was carried out by presenting the results of the evaluations from phase D, followed by a group discussion. There are four groups of participants in this study. Groups A and B are experienced end users of PGMeL and participated in phase B. Group C are users who undertook e-learning and were asked to complete the survey in phase D. Group D are the authors of the e-learning modules described above.

Results

From a list of 36 items, we developed a postgraduate Medical E-Learning Evaluation Survey (MEES). Seven residents participated in the phase B group discussion: 4 items were interpreted differently, 3 were not readable, and 2 items were double. The items from phase B were rewritten and, after adjustment, understood correctly. The MEES was translated into Dutch and again pilot-tested. All items were clear and were understood correctly. The MEES version used for the evaluation contained 3 positive domains (motivation, learning enhancers, and real-world translation) and 2 negative domains (barriers and learning discouragers), with 36 items in those domains, 5 Likert scale questions of 1 to 10, and 5 open questions asking participants to give their own comments in each domain. Three e-learning modules were evaluated from July to November 2018. There were a total of 158 responses from a Dutch module, a European OB/GYN (obstetrics and gynecology) module, and a surgical module offered worldwide. Finally, 3 focus group discussions took place with a total of 10 participants. Usefulness was much appreciated, understandability was good, and added value was high. Four items needed additional explanation by the authors, and a Creators' Manual was written at their request.

Conclusions

The MEES is the first survey to evaluate the instructional design of PGMeL and was constructed following all 7 steps of the AMEE. This study completes the design of the survey and shows its usefulness and added value to the authors. It finishes with a final, publicly available survey that includes a Creators' Manual. We briefly discuss the number of responses needed and conclude that more is better; in the end, however, one has to work with what is available. The next steps would be to see whether improvement can be measured by using the MEES and continue to work on the end understandability in different languages and cultural groups.

SUBMITTER: de Leeuw RA

PROVIDER: S-EPMC6713039 | biostudies-literature | 2019 Aug

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Development of an Instructional Design Evaluation Survey for Postgraduate Medical E-Learning: Content Validation Study.

de Leeuw Robert Adrianus RA Westerman Michiel M Walsh Kieran K Scheele Fedde F

Journal of medical Internet research 20190809 8

<h4>Background</h4>E-Learning has taken a firm place in postgraduate medical education. Whereas 10 years ago it was promising, it now has a definite niche and is clearly here to stay. However, evaluating the effect of postgraduate medical e-learning (PGMeL) and improving upon it can be complicated. While the learning aims of e-learning are evaluated, there are no instruments to evaluate the instructional design of PGMeL. Such an evaluation instrument may be developed by following the Association ...[more]

PMID: 31400102

Similar Datasets

Project description:BackgroundMedical postgraduates' demand for data capabilities is growing, as biomedical research becomes more data driven, integrative, and computational. In the context of the application of big data in health and medicine, the integration of data mining skills into postgraduate medical education becomes important.ObjectiveThis study aimed to demonstrate the design and implementation of a medical data mining course for medical postgraduates with diverse backgrounds in a medical school.MethodsWe developed a medical data mining course called "Practical Techniques of Medical Data Mining" for postgraduate medical education and taught the course online at Peking Union Medical College (PUMC). To identify the background knowledge, programming skills, and expectations of targeted learners, we conducted a web-based questionnaire survey. After determining the instructional methods to be used in the course, three technical platforms-Rain Classroom, Tencent Meeting, and WeChat-were chosen for online teaching. A medical data mining platform called Medical Data Mining - R Programming Hub (MedHub) was developed for self-learning, which could support the development and comprehensive testing of data mining algorithms. Finally, we carried out a postcourse survey and a case study to demonstrate that our online course could accommodate a diverse group of medical students with a wide range of academic backgrounds and programming experience.ResultsIn total, 200 postgraduates from 30 disciplines participated in the precourse survey. Based on the analysis of students' characteristics and expectations, we designed an optimized course structured into nine logical teaching units (one 4-hour unit per week for 9 weeks). The course covered basic knowledge of R programming, machine learning models, clinical data mining, and omics data mining, among other topics, as well as diversified health care analysis scenarios. Finally, this 9-week course was successfully implemented in an online format from May to July in the spring semester of 2020 at PUMC. A total of 6 faculty members and 317 students participated in the course. Postcourse survey data showed that our course was considered to be very practical (83/83, 100% indicated "very positive" or "positive"), and MedHub received the best feedback, both in function (80/83, 96% chose "satisfied") and teaching effect (80/83, 96% chose "satisfied"). The case study showed that our course was able to fill the gap between student expectations and learning outcomes.ConclusionsWe developed content for a data mining course, with online instructional methods to accommodate the diversified characteristics of students. Our optimized course could improve the data mining skills of medical students with a wide range of academic backgrounds and programming experience.

Project description:Objective: In order to evaluate the quality of postgraduate medical education in Germany, we examined how the learning of theoretical and practical competencies is conceptualized and how the learning process takes place in real terms. The training conditions, as perceived by medical residents, are compared with the learning objectives, as stated by the Federal Chamber of Physicians in its regulation on postgraduate education. The analysis is based on the data of the "KarMed" study (abbreviation of "career progression and career breaks among physicians during postgraduate education"), a multicentre cohort study of medical school graduates from seven universities who arre followed until they receive their licenses as specialist physicians. The study was conducted by the Institute of Primary Medical Care of the University Medical Center Hamburg-Eppendorf starting in 2008. Methodology: The KarMed study is based on annual, standardized surveys of the population described above. 48% responded in the initial survey (n=1012) and 85% or more in each of the following surveys. Descriptive statistics and logistical regressions were used for analysis. Gender-specific analyses were performed where possible in order to highlight the differences in the professional objectives and workloads between female and male physicians. Results: The study shows that both the practical and the theoretical components of postgraduate medical education in Germany are insufficient. There is a lack of a curriculum with precise learning objectives and descriptions of the corresponding educational settings. In fact, the act of learning is identical to daily clinical work. There is no structuring of the work process with regard to learning; for example, documentation procedures and feedback discussions are prescribed but largely omitted. Evidence-based medicine is not a systematic part of the training, nor is the evaluation of residents' progress. The summative final oral examination pretends that the necessary specialist competencies can be evaluated within 30 minutes. Many factors indicate that female doctors, especially those with children, have fewer learning opportunities than male doctors. Conclusion: The quality of postgraduate medical education in Germany has become inadequate, especially in an international comparison. The deficits are well known. The responsible institutions are called upon to implement sustainable reforms in the sense of postgraduate education that is structured according to educational principles and whose quality is assured.

Project description:BackgroundDigital education tools (e-learning, technology-enhanced learning) can be defined as any educational intervention that is electronically mediated. Decveloping and applying such tools and interventions for postgraduate medical professionals who work and learn after graduation can be called postgraduate medical digital education (PGMDE), which is increasingly being used and evaluated. However, evaluation has focused mainly on reaching the learning goals and little on the design. Design models for digital education (instructional design models) help educators create a digital education curriculum, but none have been aimed at PGMDE. Studies show the need for efficient, motivating, useful, and satisfactory digital education.ObjectiveOur objective was (1) to create an empirical instructional design model for PGMDE founded in evidence and theory, with postgraduate medical professionals who work and learn after graduation as the target audience, and (2) to compare our model with existing models used to evaluate and create PGMDE.MethodsPreviously we performed an integrative literature review, focus group discussions, and a Delphi procedure to determine which building blocks for such a model would be relevant according to experts and users. This resulted in 37 relevant items. We then used those 37 items and arranged them into chronological steps. After we created the initial 9-step plan, we compared these steps with other models reported in the literature.ResultsThe final 9 steps were (1) describe who, why, what, (2) select educational strategies, (3) translate to the real world, (4) choose the technology, (5) complete the team, (6) plan the budget, (7) plan the timing and timeline, (8) implement the project, and (9) evaluate continuously. On comparing this 9-step model with other models, we found that no other was as complete, nor were any of the other models aimed at PGMDE.ConclusionsOur 9-step model is the first, to our knowledge, to be based on evidence and theory building blocks aimed at PGMDE. We have described a complete set of evidence-based steps, expanding a 3-domain model (motivate, learn, and apply) to an instructional design model that can help every educator in creating efficient, motivating, useful, and satisfactory PGMDE. Although certain steps are more robust and have a deeper theoretical background in current research (such as education), others (such as budget) have been barely touched upon and should be investigated more thoroughly in order that proper guidelines may also be provided for them.

Project description:BackgroundElectronic learning (e-learning) in postgraduate medical education has seen a rapid evolution; however, we tend to evaluate it only on its primary outcome or learning aim, whereas its effectiveness also depends on its instructional design. We believe it is important to have an overview of all the methods currently used to evaluate e-learning design so that the preferred method may be identified and the next steps needed to continue to evaluate postgraduate medical e-learning may be outlined.ObjectiveThis study aimed to identify and compare the outcomes and methods used to evaluate postgraduate medical e-learning.MethodsWe performed a systematic literature review using the Web of Science, PubMed, Education Resources Information Center, and Cumulative Index of Nursing and Allied Health Literature databases. Studies that used postgraduates as participants and evaluated any form of e-learning were included. Studies without any evaluation outcome (eg, just a description of e-learning) were excluded.ResultsThe initial search identified 5973 articles, of which we used 418 for our analysis. The types of studies were trials, prospective cohorts, case reports, and reviews. The primary outcomes of the included studies were knowledge, skills, and attitude. A total of 12 instruments were used to evaluate a specific primary outcome, such as laparoscopic skills or stress related to training. The secondary outcomes mainly evaluated satisfaction, motivation, efficiency, and usefulness. We found 13 e-learning design methods across 19 studies (4% 19/418). The methods evaluated usability, motivational characteristics, and the use of learning styles or were based on instructional design theories, such as Gagne's instructional design, the Heidelberg inventory, Kern's curriculum development steps, and a scale based on the cognitive load theory. Finally, 2 instruments attempted to evaluate several aspects of a design, based on the experience of creating e-learning.ConclusionsEvaluating the effect of e-learning design is complicated. Given the diversity of e-learning methods, there are many ways to carry out such an evaluation, and probably, many ways to do so correctly. However, the current literature shows us that we have yet to reach any form of consensus about which indicators to evaluate. There is a great need for an evaluation tool that is properly constructed, validated, and tested. This could be a more homogeneous way to compare the effects of e-learning and for the authors of e-learning to continue to improve their product.

Project description:BACKGROUND:Sexual health problems are common. Therefore, training in sexual health is relevant for the clinical practice of trainees and early-career specialists in several specialties who deal with patients with sexual health problems. However, little is known about how sexual health training is provided across countries and specialties. AIM:To assess (i) sexual health training during postgraduate training programs in psychiatry, obstetrics and gynecology, urology, and endocrinology across Europe; (ii) the confidence of trainees and early-career specialists in dealing with patients with sexual health problems; and (iii) their need for further training in sexual health during postgraduate training programs. METHODS:The study was based on a collaboration among European societies of trainees in these 4 specialties. An online survey was developed and conducted from January 2015 through June 2016. MAIN OUTCOME MEASURES:Self-reported questionnaire. RESULTS:We collected 366 completed surveys from 40 countries. Sexual health training was considered an important or very important part of specialty training by 78.7% of participants. Overall, 62.3% of participants had not received any training in sexual health. Especially in obstetrics and gynecology, the large majority did not have training in sexual health (82.8%), followed by psychiatry (59.8%), urology (58.4%), and endocrinology (56.1%). There were statistically significant differences among specialties in the confidence of participants in managing patients with sexual health problems. In general, trainees and early-career specialists who had received sexual health training felt more confident in dealing with patients with gender dysphoria (P = .011), need for sexual therapy (P = .0004), paraphilic disorders (P = .0003), and sexual dysfunction (P = .0017). CONCLUSIONS:Trainees and early-career specialists found sexual health training important for their future medical practice; however, less than half received it during their postgraduate training. Participants felt more confident in managing patients with sexual health problems when sexual health training was included in the postgraduate training program. Kristufkova A, Pinto Da Costa M, Mintziori G, et al. Sexual Health During Postgraduate Training-European Survey Across Medical Specialties. Sex Med 2018;6:255-262.

Project description:IntroductionCurrent instruments to evaluate the postgraduate medical educational environment lack theoretical frameworks and are relatively long, which may reduce response rates. We aimed to develop and validate a brief instrument that, based on a solid theoretical framework for educational environments, solicits resident feedback to screen the postgraduate medical educational environment quality.MethodsStepwise, we developed a screening instrument, using existing instruments to assess educational environment quality and adopting a theoretical framework that defines three educational environment domains: content, atmosphere and organization. First, items from relevant existing instruments were collected and, after deleting duplicates and items not specifically addressing educational environment, grouped into the three domains. In a Delphi procedure, the item list was reduced to a set of items considered most important and comprehensively covering the three domains. These items were triangulated against the results of semi-structured interviews with 26 residents from three teaching hospitals to achieve face validity. This draft version of the Scan of Postgraduate Educational Environment Domains (SPEED) was administered to residents in a general and university hospital and further reduced and validated based on the data collected.ResultsTwo hundred twenty-three residents completed the 43-item draft SPEED. We used half of the dataset for item reduction, and the other half for validating the resulting SPEED (15 items, 5 per domain). Internal consistencies were high. Correlations between domain scores in the draft and brief versions of SPEED were high (>0.85) and highly significant (p<0.001). Domain score variance of the draft instrument was explained for ≥80% by the items representing the domains in the final SPEED.ConclusionsThe SPEED comprehensively covers the three educational environment domains defined in the theoretical framework. Because of its validity and brevity, the SPEED is promising as useful and easily applicable tool to regularly screen educational environment quality in postgraduate medical education.

Project description:BackgroundResidents need to be trained across the boundaries of their own specialty to prepare them for collaborative practice. Intraprofessional learning (i.e. between individuals of different disciplines within the same profession) has received little attention in the postgraduate medical education literature, in contrast to the extensive literature on interprofessional learning between individuals of different professions. To address this gap, we performed a scoping review to investigate what and how residents learn from workplace-related intraprofessional activities, and what factors influence learning.MethodsThe PRISMA guidelines were used to conduct a scoping review of empirical studies on intraprofessional workplace learning in postgraduate medical education published between 1 January 2000 to 16 April 2020 in Pubmed, Embase, PsycINFO, ERIC and Web of Science. This study applied 'best fit' framework-based synthesis to map the existing evidence, using the presage-process-product (3P) model developed by Tynjälä (2013).ResultsFour thousand three hundred thirty records were screened, and 37 articles were included. This review identified influencing (presage) factors that derived from the sociocultural environment, learner and learning context. Studies described that complexity of care can both facilitate and hinder learning. Furthermore, intraprofessional learning is threatened by professional stereotyping and negative perceptions, and awareness of learning opportunities and explicit reflection are critical in intraprofessional workplace learning. Studies described a range of informal and formal intraprofessional activities (process) under the headings of collaboration in clinical practice, rotations or placements, formal educational sessions and simulated workplace training. In general, learners responded well and their attitudes and perceptions improved, learners reported increased knowledge and skills and positive behavioural changes (product). Learning outcomes were reported in the domains of patient-centred care, collaborative attitudes and respect, mutual knowledge and understanding, collaborative decision making, communication, leadership, teamwork and reflexivity.ConclusionsThis review gives insight into the high learning potential of intraprofessional activities. Many of the included studies relied on self-reported perceptions of change, therefore, future research should focus on generating more robust evidence including objectively examined outcome measures. This review offers a comprehensive overview of the factors that influence intraprofessional workplace learning in postgraduate medical education. Finally, we provide recommendations for enhancing intraprofessional learning in clinical practice.

Project description:BackgroundTeamwork between clinical teachers is a challenge in postgraduate medical training. Although there are several instruments available for measuring teamwork in health care, none of them are appropriate for teaching teams. The aim of this study is to develop an instrument (TeamQ) for measuring teamwork, to investigate its psychometric properties and to explore how clinical teachers assess their teamwork.MethodTo select the items to be included in the TeamQ questionnaire, we conducted a content validation in 2011, using a Delphi procedure in which 40 experts were invited. Next, for pilot testing the preliminary tool, 1446 clinical teachers from 116 teaching teams were requested to complete the TeamQ questionnaire. For data analyses we used statistical strategies: principal component analysis, internal consistency reliability coefficient, and the number of evaluations needed to obtain reliable estimates. Lastly, the median TeamQ scores were calculated for teams to explore the levels of teamwork.ResultsIn total, 31 experts participated in the Delphi study. In total, 114 teams participated in the TeamQ pilot. The median team response was 7 evaluations per team. The principal component analysis revealed 11 factors; 8 were included. The reliability coefficients of the TeamQ scales ranged from 0.75 to 0.93. The generalizability analysis revealed that 5 to 7 evaluations were needed to obtain internal reliability coefficients of 0.70. In terms of teamwork, the clinical teachers scored residents' empowerment as the highest TeamQ scale and feedback culture as the area that would most benefit from improvement.ConclusionsThis study provides initial evidence of the validity of an instrument for measuring teamwork in teaching teams. The high response rates and the low number of evaluations needed for reliably measuring teamwork indicate that TeamQ is feasible for use by teaching teams. Future research could explore the effectiveness of feedback on teamwork in follow up measurements.

Dataset Information

Development of an Instructional Design Evaluation Survey for Postgraduate Medical E-Learning: Content Validation Study.

Background

Objective

Methods

Results

Conclusions

Publications

Development of an Instructional Design Evaluation Survey for Postgraduate Medical E-Learning: Content Validation Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets