Dataset Information

Rater training for standardised assessment of Objective Structured Clinical Examinations in rural Tanzania.

ABSTRACT:

Objectives

To describe a simulation-based rater training curriculum for Objective Structured Clinical Examinations (OSCEs) for clinician-based training for frontline staff caring for mothers and babies in rural Tanzania.

Background

Rater training for OSCE evaluation is widely embraced in high-income countries but not well described in low-income and middle-income countries. Helping Babies Breathe, Essential Care for Every Baby and Bleeding after Birth are standardised training programmes that encourage OSCE evaluations. Studies examining the reliability of assessments are rare.

Methods

Training of raters occurred over 3 days. Raters scored selected OSCEs role-played using standardised learners and low-fidelity mannikins, assigning proficiency levels a priori. Researchers used Zabar's criteria to critique rater agreement and mitigate measurement error during score review. Descriptive statistics, Fleiss' kappa and field notes were used to describe results.

Results

Six healthcare providers scored 42 training scenarios. There was moderate rater agreement across all OSCEs (?=0.508). Kappa values increased with Helping Babies Breathe (?=0.28-0.48) and Essential Care for Every Baby (?=0.42-0.77) by day 3 of training, but not with Bleeding after Birth (?=0.58-0.33). Raters identified average proficiency 50% of the time.

Conclusion

Our study shows that the in-country raters in this study had a hard time identifying average performance despite moderate rater agreement. Rater training is critical to ensure that the potential of training programmes translates to improved outcomes for mothers and babies; more research into the concepts and training for discernment of competence in this setting is necessary.

SUBMITTER: Sigalet EL

PROVIDER: S-EPMC7722831 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Rater training for standardised assessment of Objective Structured Clinical Examinations in rural Tanzania.

Sigalet Elaine L EL Matovelo Dismas D Brenner Jennifer L JL Boniphace Maendeleo M Ndaboine Edgar E Mwaikasu Lusako L Shabani Girles G Kabirigi Julieth J Mannerfeldt Jaelene J Singhal Nalini N

BMJ paediatrics open 20201207 1

<h4>Objectives</h4>To describe a simulation-based rater training curriculum for Objective Structured Clinical Examinations (OSCEs) for clinician-based training for frontline staff caring for mothers and babies in rural Tanzania.<h4>Background</h4>Rater training for OSCE evaluation is widely embraced in high-income countries but not well described in low-income and middle-income countries. Helping Babies Breathe, Essential Care for Every Baby and Bleeding after Birth are standardised training pro ...[more]

PMID: 33324763

Similar Datasets

Project description:BackgroundCompetence in transthoracic echocardiography (TTE) is unrelated to traditional measures of TTE competence, such as duration of training and number of examinations performed. This study aims to explore aspects of validity of an instrument for structured assessment of echocardiographic technical skills.MethodsThe study included 45 physicians with three different clinical levels of echocardiography competence who all scanned the same healthy male following national guidelines. An expert in echocardiography (OG) evaluated all the recorded, de-identified TTE images blindly using the developed instrument for assessment of TTE technical skills. The instrument consisted of both a global rating scale and a procedure specific checklist. Two scores were calculated for each examination: A global rating score and a total checklist score. OG rated ten examinations twice for intra-rater reliability, and another expert rated the same ten examinations for inter-rater reliability. A small pilot study was then performed with focus on content validity. This pilot study included nine physicians who scanned three patients with different pathologies as well as different technical difficulties.ResultsValidity of the TTE technical skills assessment instrument was supported by a significant correlation found between level of expertise and both the global score (Spearman 0.76, p<0.0001) and the checklist score (Spearman 0.74, p<0.001). Both scores were able to distinguish between the three levels of competence that were represented in the physician group. Reliability was supported by acceptable inter- and intra-rater values. The pilot study showed a tendency to improved scores with increasing expertise levels, suggesting that the instrument could also be used when pathologies were present.ConclusionsWe designed and developed a structured assessment instrument of echocardiographic technical skills that showed evidence of validity in terms of high correlations between test scores on a normal person and the level of physician competence, as well as acceptable inter- and intra-rater reliability scores. Further studies should, however, be performed to determine the adequate number of assessments needed to ensure high content validity and reliability in a clinical setting.

Project description:BackgroundTeaching and assessment of communication skills have become essential in medical education. The Objective Structured Clinical Examination (OSCE) has been found as an appropriate means to assess communication skills within medical education. Studies have demonstrated the importance of a valid assessment of medical students' communication skills. Yet, the validity of the performance scores depends fundamentally on the quality of the rating scales used in an OSCE. Thus, this systematic review aimed at providing an overview of existing rating scales, describing their underlying definition of communication skills, determining the methodological quality of psychometric studies and the quality of psychometric properties of the identified rating scales.MethodsWe conducted a systematic review to identify psychometrically tested rating scales, which have been applied in OSCE settings to assess communication skills of medical students. Our search strategy comprised three databases (EMBASE, PsycINFO, and PubMed), reference tracking and consultation of experts. We included studies that reported psychometric properties of communication skills assessment rating scales used in OSCEs by examiners only. The methodological quality of included studies was assessed using the COnsensus based Standards for the selection of health status Measurement INstruments (COSMIN) checklist. The quality of psychometric properties was evaluated using the quality criteria of Terwee and colleagues.ResultsData of twelve studies reporting on eight rating scales on communication skills assessment in OSCEs were included. Five of eight rating scales were explicitly developed based on a specific definition of communication skills. The methodological quality of studies was mainly poor. The psychometric quality of the eight rating scales was mainly intermediate.DiscussionOur results reveal that future psychometric evaluation studies focusing on improving the methodological quality are needed in order to yield psychometrically sound results of the OSCEs assessing communication skills. This is especially important given that most OSCE rating scales are used for summative assessment, and thus have an impact on medical students' academic success.

Dataset Information

Rater training for standardised assessment of Objective Structured Clinical Examinations in rural Tanzania.

Objectives

Background

Methods

Results

Conclusion

Publications

Rater training for standardised assessment of Objective Structured Clinical Examinations in rural Tanzania.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure