Dataset Information

Examining response process validity of script concordance testing: a think-aloud approach.

ABSTRACT:

Objectives

This study investigated whether medical student responses to Script Concordance Testing (SCT) items represent valid clinical reasoning. Using a think-aloud approach students provided written explanations of the reasoning that underpinned their responses, and these were reviewed for concordance with an expert reference panel.

Methods

A set of 12, 11 and 15 SCT items were administered online to Year 3 (2018), Year 4 (2018) and Year 3 (2019) medical students respectively. Students' free-text descriptions of the reasoning supporting each item response were analysed, and compared with those of the expert panel. Response process validity was quantified as the rate of true positives (percentage of full and partial credit responses derived through correct clinical reasoning); and true negatives (percentage of responses with no credit derived through faulty clinical reasoning).

Results

Two hundred and nine students completed the online tests (response rate = 68.3%). The majority of students who had chosen the response which attracted full or partial credit also provided justifications which were concordant with the experts (true positive rate of 99.6% for full credit; 99.4% for partial credit responses). Most responses that attracted no credit were based on faulty clinical reasoning (true negative of 99.0%).

Conclusions

The findings provide support for the response process validity of SCT scores in the setting of undergraduate medicine. The additional written think-aloud component, to assess clinical reasoning, provided useful information to inform student learning. However, SCT scores should be validated on each testing occasion, and in other contexts.

SUBMITTER: Wan MSH

PROVIDER: S-EPMC7870454 | biostudies-literature | 2020 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Examining response process validity of script concordance testing: a think-aloud approach.

Wan Michael Siu Hong MSH Tor Elina E Hudson Judith N JN

International journal of medical education 20200624

<h4>Objectives</h4>This study investigated whether medical student responses to Script Concordance Testing (SCT) items represent valid clinical reasoning. Using a think-aloud approach students provided written explanations of the reasoning that underpinned their responses, and these were reviewed for concordance with an expert reference panel.<h4>Methods</h4>A set of 12, 11 and 15 SCT items were administered online to Year 3 (2018), Year 4 (2018) and Year 3 (2019) medical students respectively. ...[more]

PMID: 32581143

Similar Datasets

Project description:BackgroundSituational judgment tests (SJTs) are used in health sciences education to measure examinee knowledge using case-based scenarios. Despite their popularity, there is a significant gap in the validity research on the response process that demonstrates how SJTs measure their intended constructs. A model of SJT response processes has been proposed in the literature by Robert Ployhart; however, few studies have explored and expanded the factors. The purpose of this study was to describe the factors involved in cognitive processes that examinees use as they respond to SJT items in a health professions education context.MethodsThirty participants-15 student pharmacists and 15 practicing pharmacists-completed a 12-item SJT designed to measure empathy. Each participant engaged in a think-aloud interview while completing the SJT, followed by a cognitive interview probing their decision-making processes. Interviews were transcribed and independently coded by three researchers to identify salient factors that contributed to response processes.ResultsThe findings suggest SJT response processes include all four stages (comprehension, retrieval, judgment, and response selection) as initially proposed by Ployhart. The study showed factors from other published research were present, including job-specific knowledge and experiences, emotional intelligence, and test-taking. The study also identified new factors not yet described, including identifying a task objective in the scenario, assumptions about the scenario, perceptions about the scenario, and the setting of the item.ConclusionsThis study provides additional SJT validity evidence by exploring participants' response processes through cognitive and think-aloud interviews. It also confirmed the four-stage model previously described by Ployhart and identified new factors that may influence SJT response processes. This study contributes to the literature with an expanded SJT response process model in a health professions education context and offers an approach to evaluate SJT response processes in the future.

Project description:ObjectivesTo determine the response process validity, feasibility of completion, acceptability and preferences for three patient-reported outcome measures that could be used in economic evaluation-the EQ-5D-5L, ICECAP-A and ICECAP-O-in people requiring kidney care.DesignParticipants were asked to 'think-aloud' while completing the EQ-5D-5L, ICECAP-A and ICECAP-O, followed by a semistructured interview. Five raters identified errors or struggles in completing the measures from the think-aloud component of the transcripts. Patient preferences for measures were extracted from the semistructured interview.SettingEligible patients were identified through a large UK secondary care renal centre.ParticipantsIn total, 30 participants were included in the study, consisting of patients attending renal outpatients for chronic kidney disease (n=18), with a functioning kidney transplant (n=6) and receiving haemodialysis (n=6).ResultsParticipants had few errors and struggles in completing the EQ-5D-5L (11% error rate, 3% struggle rate), ICECAP-A (2% error rate, 2% struggle rate) and ICECAP-O (4% error rate, 3% struggle rate). The main errors with the EQ-5D-5L were judgements that did not comply with the 'your health today' instruction. Comprehension errors were most prominent on ICECAP-O. Judgement errors were the only errors reported on ICECAP-A. Although the EQ-5D-5L had slightly more errors and struggles, it was the measure most preferred, with participants able to make a clearer link with EQ-5D-5L and their health condition.ConclusionsThe EQ-5D-5L, ICECAP-A and ICECAP-O are feasible for people requiring kidney care to complete and can be included in studies conducting economic evaluations of kidney care interventions. Further research is required to assess how health (eg, EQ-5D) and capability (eg, ICECAP) measures can be included in an economic evaluation simultaneously, as well as what ICECAP measure(s) to include when patient groups straddle the age ranges for ICECAP-A (18 years and older) and ICECAP-O (65 years and older).

Project description:BackgroundSpending time in natural environments is beneficial for human health, but many older adults have limited or no access to natural environments. Virtual reality technology may be a means to facilitate nature experiences, and so, there is a need for knowledge on how to design virtual restorative natural environments for older adults.ObjectiveThe aim of this study was to identify, implement, and test older adults' preferences and ideas regarding virtual natural environments.MethodsA total of 14 older adults (mean age 75, SD 5.9 years) participated in an iterative process to design such an environment. We used think-aloud protocols and qualitative content analysis and established questionnaires that targeted usability, affective aspects, and side effects. These data guided the design decisions for incremental implementations of a prototype.ResultsThe participants' preferences included trueness to reality in terms of rendition and behavior; traces of human activity and natural processes that trigger the imagination and provide believability; the ability to roam, explore, and interact with the environment; and a familiar, relatable environment that evokes memories. The iterative design process resulted in a prototype featuring many of the participants' ideas and preferences, including a seated locomotion technique, animals, a boat ride, the discovery of a boat wreck, and apple picking. The questionnaire results indicated high perceived usability, interest, and enjoyment; low pressure and tension; moderate value and usefulness; and negligible side effects.ConclusionsWe suggested 3 principles for virtual natural environments for older adults: realness, interactivity, and relatedness. Virtual natural environments should also provide a diversity of content and activities to accommodate the heterogeneity in older adults' preferences. These results can contribute to a framework for designing virtual natural environments for older adults. However, these findings need to be tested and potentially revised in future studies.

Project description:BackgroundPatients with cancer require adequate preparation in self-management of treatment toxicities to reduce morbidity that can be achieved through well-designed digital technologies that are developed in co-design with patients and end users.ObjectiveWe undertook a user-centered co-design process in partnership with patients and other knowledge end users to develop and iteratively test an evidence-based and theoretically informed web-based cancer self-management program (I-Can Manage). The specific study aims addressed in 2 phases were to (1) identify from the perspective of patients with cancer and clinicians the desired content, features, and functionalities for an online self-management education and support (SMES) program to enable patient self-management of treatment toxicities (phase 1); (2) develop the SMES prototype based on human-centered, health literate design principles and co-design processes; and (3) evaluate usability of the I-Can Manage prototype through user-centered testing (phase 2).MethodsWe developed the I-Can Manage program using multiperspective data sources and based on humanistic and co-design principles with end users engaged through 5 phases of development. We recruited adult patients with lung, colorectal, and lymphoma cancer receiving systemic treatments from ambulatory clinics in 2 regional cancer programs for the qualitative inquiry phase. The design of the program was informed by data from qualitative interviews and focus groups, persona and journey mapping, theoretical underpinnings of social cognitive learning theory, and formalized usability testing using a cognitive think-aloud process and user satisfaction survey. A co-design team comprising key stakeholders (human design experts, patients/caregiver, clinicians, knowledge end users, and e-learning and digital design experts) was involved in the developmental process. We used a cognitive think-aloud process to test usability and participants completed the Post-Study System Usability Questionnaire (PSSUQ).ResultsIn the initial qualitative inquiry phase, 16 patients participated in interviews and 19 clinicians participated in interviews or focus groups and 12 key stakeholders participated in a persona journey mapping workshop to inform development of the program prototype. The I-Can Manage program integrates evidence-based information and strategies for the self-management of treatment toxicities and health-promoting behaviors in 6 e-learning modules (lay termed "chapters"), starting with an orientation to self-management. Behavioral exercises, patient written and video stories, downloadable learning resources, and online completion of goals and action plans were integrated across chapters. Patient participants (n=5) with different cancers, gender, and age worked through the program in the human factors laboratory using a cognitive think-aloud process and all key stakeholders reviewed each chapter of the program and approved revisions. Results of the PSSUQ (mean total score: 3.75) completed following the cognitive think-aloud process (n=5) suggest patient satisfaction with the usability of I-Can Manage.ConclusionsThe I-Can Manage program has the potential for activating patients in self-management of cancer and treatment toxicities but requires testing in a larger randomized controlled trial.

Dataset Information

Examining response process validity of script concordance testing: a think-aloud approach.

Objectives

Methods

Results

Conclusions

Publications

Examining response process validity of script concordance testing: a think-aloud approach.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets