Dataset Information

Development and validation of a portfolio assessment system for medical schools in Korea.

ABSTRACT:

Purpose

Consistent evaluation procedures based on objective and rational standards are essential for the sustainability of portfolio-based education, which has been widely introduced in medical education. We aimed to develop and implement a portfolio assessment system, and to assess its validity and reliability.

Methods

We developed a portfolio assessment system from March 2019 to August 2019 and confirmed its content validity through expert assessment by an expert group comprising 2 medical education specialists, 2 professors involved in education at medical school, and a professor of basic medical science. Six trained assessors conducted 2 rounds of evaluation of 7 randomly selected portfolios for the “Self-Development and Portfolio II” course from January 2020 to July 2020. These data are used inter-rater reliability was evaluated using intra-class correlation coefficients (ICCs) in September 2020.

Results

The portfolio assessment system is based on the following process; assessor selection, training, analytical/comprehensive evaluation, and consensus. Appropriately trained assessors evaluated portfolios based on specific assessment criteria and a rubric for assigning points. In the analysis of inter-rater reliability, the first round of evaluation grades was submitted, and all assessment areas except “goal-setting” showed a high ICC of 0.81 or higher. After the first round of assessment, we attempted to standardize objective assessment procedures. As a result, all components of the assessments showed close correlations, with ICCs of 0.81 or higher.

Conclusion

We confirmed that when assessors with an appropriate training conduct portfolio assessment based on specified standards through a systematic procedure, the results are reliable.

SUBMITTER: Yoo DM

PROVIDER: S-EPMC7859386 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Development and validation of a portfolio assessment system for medical schools in Korea.

Yoo Dong Mi DM Cho A Ra AR Kim Sun S

Journal of educational evaluation for health professions 20201209

<h4>Purpose</h4>Consistent evaluation procedures based on objective and rational standards are essential for the sustainability of portfolio-based education, which has been widely introduced in medical education. We aimed to develop and implement a portfolio assessment system, and to assess its validity and reliability.<h4>Methods</h4>We developed a portfolio assessment system from March 2019 to August 2019 and confirmed its content validity through expert assessment by an expert group comprisin ...[more]

PMID: 33291206

Similar Datasets

Project description:BackgroundSchool health programs need to target all aspects of physical, psychological, and social well-being. Using a slightly modified version of the COSMIN Risk of Bias checklist, we developed and conducted the first validation of the School Health Assessment Tool for Primary Schools (SHAT-PS).MethodThe exploratory sequential mixed method was used in this study. In the first phase, scientific databases were systematically searched to find school health models and instruments and 65 interviews were conducted with school stakeholders. The Colaizzi's method was used to code the qualitative data into themes. Then, a pool of items was created for each theme, rechecked by psychometric experts and then validated for content (i.e., relevance, clarity, and comprehensiveness) by psychometric experts and individuals of the target population (i.e., school personnel). In the second phase, classical test theory was utilized to analyze the validity and reliability of the resulting items from phase 1 among 400 individuals working at primary schools.ResultsThe coding of the interviews resulted in ten themes that we labeled based on the theoretical literature: school health policies, community connections, health education, physical activity, health services, nutrition, psychological services, physical environment, equipment and facilities, and school staff's health. The items created for each theme ended up in an initial pool of 76 items. In the final stage of phase 1, 69 items remained after the content validity assessment by experts and school personnel. In phase 2, the SHAT-PS items were tested using maximum likelihood exploratory factor analysis and confirmatory factor analysis. Of the 69 items from phase 1, 22 items were removed due to low factor loadings. The results showed that the 8-factor model was the best solution (chi-square/df = 2.41, CFI = .98, TLI = .97, RMSEA = .06). The discriminant and convergent validity of the SHAT-PS were evaluated as satisfactory and the scale had high internal consistency (Cronbach's alpha for all subscales > .93). The test-retest reliability was satisfactory-the intraclass correlation coefficient pooled was .95 (99% CI [.91-.98]). Moreover, the standard error of measurement resulted in an SEM pooled equal to 4.4. No discrepancy was found between subgroups of gender and subgroups of staffs' positions at schools.ConclusionThe SHAT-PS is a valid and reliable tool that may facilitate school staff, stakeholders and researchers to evaluate the presence of the factors that promote health at primary schools. Nevertheless, in the process of validation, many of the items related to staff's health were eliminated due to poor factor loadings. Obviously, staff health is an important factor in the measurement of school health. Hence, we recommend that the validity and reliability of the SHAT-PS in other cultures should be done using the original 76-item version.

Project description:Assessment environment, synonymous with climate or atmosphere, is multifaceted. Although there are valid and reliable instruments for measuring the educational environment, there is no validated instrument for measuring the assessment environment in medical programs. This study aimed to develop an instrument for measuring students' perceptions of the assessment environment in an undergraduate medical program and to examine the psychometric properties of the new instrument.The Assessment Environment Questionnaire (AEQ), a 40-item, four-point (1=Strongly Disagree to 4=Strongly Agree) Likert scale instrument designed by the authors, was administered to medical undergraduates from the authors' institution. The response rate was 626/794 (78.84%). To establish construct validity, exploratory factor analysis (EFA) with principal component analysis and varimax rotation was conducted. To examine the internal consistency reliability of the instrument, Cronbach's α was computed. Mean scores for the entire AEQ and for each factor/subscale were calculated. Mean AEQ scores of students from different academic years and sex were examined.Six hundred and eleven completed questionnaires were analysed. EFA extracted four factors: feedback mechanism (seven items), learning and performance (five items), information on assessment (five items), and assessment system/procedure (three items), which together explained 56.72% of the variance. Based on the four extracted factors/subscales, the AEQ was reduced to 20 items. Cronbach's α for the 20-item AEQ was 0.89, whereas Cronbach's α for the four factors/subscales ranged from 0.71 to 0.87. Mean score for the AEQ was 2.68/4.00. The factor/subscale of 'feedback mechanism' recorded the lowest mean (2.39/4.00), whereas the factor/subscale of 'assessment system/procedure' scored the highest mean (2.92/4.00). Significant differences were found among the AEQ scores of students from different academic years.The AEQ is a valid and reliable instrument. Initial validation supports its use to measure students' perceptions of the assessment environment in an undergraduate medical program.

Project description:BackgroundThe Australian Medical Council, which accredits Australian medical schools, recommends medical leadership graduate outcomes be taught, assessed and accredited. In Australia and New Zealand (Australasia) there is a significant research gap and no national consensus on how to educate, assess, and evaluate leadership skills in medical professional entry degree/programs. This study aims to investigate the current curricula, assessment and evaluation of medical leadership in Australasian medical degrees, with particular focus on the roles and responsibilities of medical leadership teachers, frameworks used and competencies taught, methods of delivery, and barriers to teaching leadership.MethodsA self-administered cross-sectional survey was distributed to senior academics and/or heads or Deans of Australasian medical schools. Data for closed questions and ordinal data of each Likert scale response were described via frequency analysis. Content analysis was undertaken on free text responses and coded manually.ResultsSixteen of the 22 eligible (73%) medical degrees completed the full survey and 100% of those indicate that leadership is taught in their degree. In most degrees (11, 69%) leadership is taught as a common theme integrated throughout the curricula across several subjects. There is a variety of leadership competencies taught, with strengths being communication (100%), evidence based practice (100%), critical reflective practice (94%), self-management (81%), ethical decision making (81%), critical thinking and decision making (81%). Major gaps in teaching were financial management (20%), strategic planning (31%) and workforce planning (31%). The teaching methods used to deliver medical leadership within the curricula are diverse, with many degrees providing opportunities for leadership teaching for students outside the curricula. Most degrees (10, 59%) assess the leadership education, with one-third (6, 35%) evaluating it.ConclusionsMedical leadership competencies are taught in most degrees, but key leadership competencies are not being taught and there appears to be no continuous quality improvement process for leadership education. There is much more we can do as medical educators, academics and leaders to shape professional development of academics to teach medical leadership, and to agree on required leadership skills set for our students so they can proactively shape the future of the health care system.

Project description:BackgroundAssessment of the quality of medical evidence available on the web is a critical step in the preparation of systematic reviews. Existing tools that automate parts of this task validate the quality of individual studies but not of entire bodies of evidence and focus on a restricted set of quality criteria.ObjectiveWe proposed a quality assessment task that provides an overall quality rating for each body of evidence (BoE), as well as finer-grained justification for different quality criteria according to the Grading of Recommendation, Assessment, Development, and Evaluation formalization framework. For this purpose, we constructed a new data set and developed a machine learning baseline system (EvidenceGRADEr).MethodsWe algorithmically extracted quality-related data from all summaries of findings found in the Cochrane Database of Systematic Reviews. Each BoE was defined by a set of population, intervention, comparison, and outcome criteria and assigned a quality grade (high, moderate, low, or very low) together with quality criteria (justification) that influenced that decision. Different statistical data, metadata about the review, and parts of the review text were extracted as support for grading each BoE. After pruning the resulting data set with various quality checks, we used it to train several neural-model variants. The predictions were compared against the labels originally assigned by the authors of the systematic reviews.ResultsOur quality assessment data set, Cochrane Database of Systematic Reviews Quality of Evidence, contains 13,440 instances, or BoEs labeled for quality, originating from 2252 systematic reviews published on the internet from 2002 to 2020. On the basis of a 10-fold cross-validation, the best neural binary classifiers for quality criteria detected risk of bias at 0.78 F1 (P=.68; R=0.92) and imprecision at 0.75 F1 (P=.66; R=0.86), while the performance on inconsistency, indirectness, and publication bias criteria was lower (F1 in the range of 0.3-0.4). The prediction of the overall quality grade into 1 of the 4 levels resulted in 0.5 F1. When casting the task as a binary problem by merging the Grading of Recommendation, Assessment, Development, and Evaluation classes (high+moderate vs low+very low-quality evidence), we attained 0.74 F1. We also found that the results varied depending on the supporting information that is provided as an input to the models.ConclusionsDifferent factors affect the quality of evidence in the context of systematic reviews of medical evidence. Some of these (risk of bias and imprecision) can be automated with reasonable accuracy. Other quality dimensions such as indirectness, inconsistency, and publication bias prove more challenging for machine learning, largely because they are much rarer. This technology could substantially reduce reviewer workload in the future and expedite quality assessment as part of evidence synthesis.

Dataset Information

Development and validation of a portfolio assessment system for medical schools in Korea.

Purpose

Methods

Results

Conclusion

Publications

Development and validation of a portfolio assessment system for medical schools in Korea.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets