Dataset Information

Derivation of Two Critical Appraisal Scores for Trainees to Evaluate Online Educational Resources: A METRIQ Study.

ABSTRACT: Online education resources (OERs), like blogs and podcasts, increasingly augment or replace traditional medical education resources such as textbooks and lectures. Trainees' ability to evaluate these resources is poor, and few quality assessment aids have been developed to assist them. This study aimed to derive a quality evaluation instrument for this purpose.We used a three-phase methodology. In Phase 1, a previously derived list of 151 OER quality indicators was reduced to 13 items using data from published consensus-building studies (of medical educators, expert podcasters, and expert bloggers) and subsequent evaluation by our team. In Phase 2, these 13 items were converted to seven-point Likert scales used by trainee raters (n=40) to evaluate 39 OERs. The reliability and usability of these 13 rating items was determined using responses from trainee raters, and top items were used to create two OER quality evaluation instruments. In Phase 3, these instruments were compared to an external certification process (the ALiEM AIR certification) and the gestalt evaluation of the same 39 blog posts by 20 faculty educators.Two quality-evaluation instruments were derived with fair inter-rater reliability: the METRIQ-8 Score (Inter class correlation coefficient [ICC]=0.30, p<0.001) and the METRIQ-5 Score (ICC=0.22, p<0.001). Both scores, when calculated using the derivation data, correlated with educator gestalt (Pearson's r=0.35, p=0.03 and r=0.41, p<0.01, respectively) and were related to increased odds of receiving an ALiEM AIR certification (odds ratio=1.28, p=0.03; OR=1.5, p=0.004, respectively).Two novel scoring instruments with adequate psychometric properties were derived to assist trainees in evaluating OER quality and correlated favourably with gestalt ratings of online educational resources by faculty educators. Further testing is needed to ensure these instruments are accurate when applied by trainees.

SUBMITTER: Chan TM

PROVIDER: S-EPMC5017842 | biostudies-literature | 2016 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Derivation of Two Critical Appraisal Scores for Trainees to Evaluate Online Educational Resources: A METRIQ Study.

Chan Teresa M TM Thoma Brent B Krishnan Keeth K Lin Michelle M Carpenter Christopher R CR Astin Matt M Kulasegaram Kulamakan K

The western journal of emergency medicine 20160726 5

<h4>Introduction</h4>Online education resources (OERs), like blogs and podcasts, increasingly augment or replace traditional medical education resources such as textbooks and lectures. Trainees' ability to evaluate these resources is poor, and few quality assessment aids have been developed to assist them. This study aimed to derive a quality evaluation instrument for this purpose.<h4>Methods</h4>We used a three-phase methodology. In Phase 1, a previously derived list of 151 OER quality indicato ...[more]

PMID: 27625722

Similar Datasets

Project description:BackgroundRoutine admission to an intensive care unit (ICU) following brain tumor surgery has been a common practice for many years. Although this practice has been challenged by many authors, it has still not changed widely, mainly due to the lack of reliable data for preoperative risk assessment. Motivated by this dilemma, risk prediction scores for postoperative complications following brain tumor surgery have been developed recently. In order to improve the ICU admission policy at our institution, we assessed the applicability, performance, and safety of the two most appropriate risk prediction scores.MethodsOne thousand consecutive adult patients undergoing elective brain tumor resection within 19 months were included. Patients with craniotomy for other causes, i.e., cerebral aneurysms and microvascular decompression, were excluded. The decision for postoperative ICU-surveillance was made by joint judgment of the operating surgeon and the anesthesiologist. All data and features relevant to the scores were extracted from clinical records and subsequent ICU or neurosurgical floor documentation was inspected for any postoperative adverse events requiring ICU admission. The CranioScore derived by Cinotti et al. (Anesthesiology 129(6):1111-20, 5) and the risk assessment score of Munari et al. (Acta Neurochir (Wien) 164(3):635-641, 15) were calculated and prognostic performance was evaluated by ROC analysis.ResultsIn our cohort, both scores showed only a weak prognostic performance: the CranioScore reached a ROC-AUC of 0.65, while Munari et al.'s score achieved a ROC-AUC of 0.67. When applying the recommended decision thresholds for ICU admission, 64% resp. 68% of patients would be classified as in need of ICU surveillance, and the negative predictive value (NPV) would be 91% for both scores. Lowering the thresholds in order to increase patient safety, i.e., 95% NPV, would lead to ICU admission rates of over 85%.ConclusionPerformance of both scores was limited in our cohort. In practice, neither would achieve a significant reduction in ICU admission rates, whereas the number of patients suffering complications at the neurosurgical ward would increase. In future, better risk assessment measures are needed.

Project description:BackgroundResearch is an essential activity for improving quality and efficiency in healthcare. The objective of this study was to train nurses from the public Basque Health Service (Osakidetza) in critical appraisal, promoting continuous training and the use of research in clinical practice.MethodsThis was a prospective pre-post test study. The InfoCritique course on critical appraisal was translated and adapted. A sample of 50 nurses and 3 tutors was recruited. Educational strategies and assessment instruments were established for the course. A course website was created that contained contact details of the teaching team and coordinator, as well as a course handbook and videos introducing the course. Assessment comprised the administration of questionnaires before and after the course, in order to explore the main intervention outcomes: knowledge acquired and self-learning readiness. Satisfaction was also measured at the end of the course.ResultsOf the 50 health professionals recruited, 3 did not complete the course for personal or work-related reasons. The mean score on the pre-course knowledge questionnaire was 70.5 out of 100, with a standard deviation of 11.96. In general, participants' performance on the knowledge questionnaire improved after the course, as reflected in the notable increase of the mean score, to 86.6, with a standard deviation of 10.00. Further, analyses confirmed statistically significant differences between pre- and post-course results (p < 0.001). With regard to self-learning readiness, after the course, participants reported a greater readiness and ability for self-directed learning. Lastly, in terms of level of satisfaction with the course, the mean score was 7 out of 10.ConclusionsParticipants significantly improved their knowledge score and self-directed learning readiness after the educational intervention, and they were overall satisfied with the course. For the health system and nursing professionals, this type of course has the potential to provide methodological tools for research, promote a research culture, and encourage critical thinking for evidence-based decision making.

Dataset Information

Derivation of Two Critical Appraisal Scores for Trainees to Evaluate Online Educational Resources: A METRIQ Study.

Publications

Derivation of Two Critical Appraisal Scores for Trainees to Evaluate Online Educational Resources: A METRIQ Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets