Dataset Information

Preliminary Validity Evidence for a Milestones-Based Rating Scale for Chart-Stimulated Recall.

ABSTRACT:

Background?

Minimally anchored Standard Rating Scales (SRSs), which are widely used in medical education, are hampered by suboptimal interrater reliability. Expert-derived frameworks, such as the Accreditation Council for Graduate Medical Education (ACGME) Milestones, may be helpful in defining level-specific anchors to use on rating scales.

Objective?

We examined validity evidence for a Milestones-Based Rating Scale (MBRS) for scoring chart-stimulated recall (CSR).

Methods?

Two 11-item scoring forms with either an MBRS or SRS were developed. Items and anchors for the MBRS were adapted from the ACGME Internal Medicine Milestones. Six CSR standardized videos were developed. Clinical faculty scored videos using either the MBRS or SRS and following a randomized crossover design. Reliability of the MBRS versus the SRS was compared using intraclass correlation.

Results?

Twenty-two faculty were recruited for instrument testing. Some participants did not complete scoring, leaving a response rate of 15 faculty (7 in the MBRS group and 8 in the SRS group). A total of 529 ratings (number of items × number of scores) using SRSs and 540 using MBRSs were available. Percent agreement was higher for MBRSs for only 2 of 11 items-use of consultants (92 versus 75, P = .019) and unique characteristics of patients (96 versus 79, P = .011)-and the overall score (89 versus 82, P < .001). Interrater agreement was 0.61 for MBRSs and 0.51 for SRSs.

Conclusions?

Adding milestones to our rating form resulted in significant, but not substantial, improvement in intraclass correlation coefficient. Improvement was inconsistent across items.

SUBMITTER: Reddy ST

PROVIDER: S-EPMC6008016 | biostudies-literature | 2018 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Preliminary Validity Evidence for a Milestones-Based Rating Scale for Chart-Stimulated Recall.

Reddy Shalini T ST Tekian Ara A Durning Steven J SJ Gupta Shanu S Endo Justin J Affinati Brenda B Park Yoon Soo YS

Journal of graduate medical education 20180601 3

<h4>Background</h4>Minimally anchored Standard Rating Scales (SRSs), which are widely used in medical education, are hampered by suboptimal interrater reliability. Expert-derived frameworks, such as the Accreditation Council for Graduate Medical Education (ACGME) Milestones, may be helpful in defining level-specific anchors to use on rating scales.<h4>Objective</h4>We examined validity evidence for a Milestones-Based Rating Scale (MBRS) for scoring chart-stimulated recall (CSR).<h4>Methods</h4>T ...[more]

PMID: 29946382

Similar Datasets

Project description:BackgroundChart-stimulated recall (CSR) is a case-based interviewing technique, which is used in the assessment of clinical decision-making in medical education and professional certification. Increasingly, clinical decision-making is a concern for clinical research in primary care. In this study, we review the prior application and utility of CSR as a technique for research interviews in primary care.MethodsFollowing Arksey & O'Malley's method for scoping reviews, we searched seven databases, grey literature, reference lists, and contacted experts in the field. We excluded studies on medical education or competence assessment. Retrieved citations were screened by one reviewer and full texts were ordered for all potentially relevant abstracts. Two researchers independently reviewed full texts and performed data extraction and quality appraisal if inclusion criteria were met. Data were collated and summarised using a published framework on the reporting of qualitative interview techniques, which was chosen a priori. The preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines informed the review report.ResultsFrom an initial list of 789 citations, eight studies using CSR in research interviews were included in the review: six from North America, one from the Netherlands, and one from Ireland. The most common purpose of included studies was to examine the influence of guidelines on physicians' decisions. The number of interviewees ranged from seven to twenty nine, while the number of charts discussed per interview ranged from one to twelve. CSR gave insights into physicians' reasoning for actions taken or not taken; the unrecorded social and clinical influences on decisions; and discrepancies between physicians' real and perceived practice. Ethical concerns and the training and influence of the researcher were poorly discussed in most of the studies. Potential pitfalls included the risk of recall, selection and observation biases.ConclusionsDespite the proven validity, reliability and acceptability of CSR in assessment interviews in medical education, its use in clinical research is limited. Application of CSR in qualitative research brings interview data closer to the reality of practice. Although further development of the approach is required, we recommend a role for CSR in research interviews on decision-making in clinical practice.

Project description:BackgroundUncertainty is common and impacts both patients and clinicians. The approach to uncertainty in medical trainees may be distinct from that of practicing clinicians and has important implications for medical education.ObjectiveDescribe trainee approach to uncertainty with the use of chart-stimulated recall (CSR)-based interviews, as well as the utility of such interviews in promoting reflection about decision-making among senior internal medicine (IM) residents.DesignQualitative analysis of CSR-based interviews with IM residents.ParticipantsSenior IM residents rotating on inpatient night float at the University of Pittsburgh Medical Center from February to September 2019.InterventionEach participant completed one, 20-min CSR session based on a self-selected case in which there was uncertainty in decision-making. Interviews explored the sources of, approaches to, and feelings about uncertainty.ApproachTwo independent coders developed a codebook and independently coded all transcripts. Transcripts were then analyzed using thematic analysis.Key resultsThe perceived acuity of the patient presentation was the main driver of the approach to and stress related to uncertainty. Perceived level of responsibility in resolving uncertainty during the overnight shift also varied among individual participants. Attending expression of uncertainty provided comfort to residents and alleviated stress related to uncertainty. Residents felt comfortable discussing their uncertainty and felt that the opportunity to think aloud during the exercise was valuable.ConclusionsOur study demonstrated a novel approach to the exploration of uncertainty in medical decision-making, with the use of CSR. Variations in resident perceived level of responsibility in resolving uncertainty during the overnight shift suggest a need for curriculum development in approach to uncertainty during night shifts. Though residents often experienced stress related to uncertainty, attending expression of uncertainty was an important mitigator of that stress, emphasizing the important role that the trainee-attending interaction plays in the diagnostic process.

Dataset Information

Preliminary Validity Evidence for a Milestones-Based Rating Scale for Chart-Stimulated Recall.

Background?

Objective?

Methods?

Results?

Conclusions?

Publications

Preliminary Validity Evidence for a Milestones-Based Rating Scale for Chart-Stimulated Recall.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets