Dataset Information

Assessing the validity of using serious game technology to analyze physician decision making.

ABSTRACT:

Background

Physician non-compliance with clinical practice guidelines remains a critical barrier to high quality care. Serious games (using gaming technology for serious purposes) have emerged as a method of studying physician decision making. However, little is known about their validity.

Methods

We created a serious game and evaluated its construct validity. We used the decision context of trauma triage in the Emergency Department of non-trauma centers, given widely accepted guidelines that recommend the transfer of severely injured patients to trauma centers. We designed cases with the premise that the representativeness heuristic influences triage (i.e. physicians make transfer decisions based on archetypes of severely injured patients rather than guidelines). We randomized a convenience sample of emergency medicine physicians to a control or cognitive load arm, and compared performance (disposition decisions, number of orders entered, time spent per case). We hypothesized that cognitive load would increase the use of heuristics, increasing the transfer of representative cases and decreasing the transfer of non-representative cases.

Findings

We recruited 209 physicians, of whom 168 (79%) began and 142 (68%) completed the task. Physicians transferred 31% of severely injured patients during the game, consistent with rates of transfer for severely injured patients in practice. They entered the same average number of orders in both arms (control (C): 10.9 [SD 4.8] vs. cognitive load (CL):10.7 [SD 5.6], p?=?0.74), despite spending less time per case in the control arm (C: 9.7 [SD 7.1] vs. CL: 11.7 [SD 6.7] minutes, p<0.01). Physicians were equally likely to transfer representative cases in the two arms (C: 45% vs. CL: 34%, p?=?0.20), but were more likely to transfer non-representative cases in the control arm (C: 38% vs. CL: 26%, p?=?0.03).

Conclusions

We found that physicians made decisions consistent with actual practice, that we could manipulate cognitive load, and that load increased the use of heuristics, as predicted by cognitive theory.

SUBMITTER: Mohan D

PROVIDER: S-EPMC4143260 | biostudies-literature | 2014

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Assessing the validity of using serious game technology to analyze physician decision making.

Mohan Deepika D Angus Derek C DC Ricketts Daniel D Farris Coreen C Fischhoff Baruch B Rosengart Matthew R MR Yealy Donald M DM Barnato Amber E AE

PloS one 20140825 8

<h4>Background</h4>Physician non-compliance with clinical practice guidelines remains a critical barrier to high quality care. Serious games (using gaming technology for serious purposes) have emerged as a method of studying physician decision making. However, little is known about their validity.<h4>Methods</h4>We created a serious game and evaluated its construct validity. We used the decision context of trauma triage in the Emergency Department of non-trauma centers, given widely accepted gui ...[more]

PMID: 25153149

Similar Datasets

Project description:Background:A serious game (SG) is a useful tool for nurse training. The objectives of this study were to assess validity evidence of a new SG designed to improve nurses' ability to detect patient clinical deterioration. Methods:The SG (LabForGames Warning) was developed through interaction between clinical and pedagogical experts and one developer. For the game study, consenting nurses were divided into three groups: nursing students (pre-graduate) (group S), recently graduated nurses (graduated < 2?years before the study) (group R) and expert nurses (graduated > 4?years before the study and working in an ICU) (group E). Each volunteer played three cases of the game (haemorrhage, brain trauma and obstructed intestinal tract). The validity evidence was assessed following Messick's framework: content, response process (questionnaire, observational analysis), internal structure, relations to other variables (by scoring each case and measuring playing time) and consequences (a posteriori analysis). Results:The content validity was supported by the game design produced by clinical, pedagogical and interprofessional experts in accordance with the French nurse training curriculum, literature review and pilot testing. Seventy-one nurses participated in the study: S (n = 25), R (n = 25) and E (n = 21). The content validity in all three cases was highly valued by group E. The response process evidence was supported by good security control. There was no significant difference in the three groups' high rating of the game's realism, satisfaction and educational value. All participants stated that their knowledge of the different steps of the clinical reasoning process had improved. Regarding the internal structure, the factor analysis showed a common source of variance between the steps of the clinical reasoning process and communication or the situational awareness errors made predominantly by students. No statistical difference was observed between groups regarding scores and playing time. A posteriori analysis of the results of final examinations assessing study-related topics found no significant difference between group S participants and students who did not participate in the study. Conclusion:While it appears that this SG cannot be used for summative assessment (score validity undemonstrated), it is positively valued as an educational tool. Trial registration:ClinicalTrials.gov ID: NCT03092440.

Project description:Shared decision making (SDM) is important in achieving patient-centered care. SDM tools such as decision aids are intended to inform the patient. When used to assist in decision making between treatments, decision aids have been shown to reduce decisional conflict, increase ease of decision making, and increase modification of previous decisions.The purpose of this systematic review is to assess the impact of computerized decision aids on patient-centered outcomes related to SDM for seriously ill patients.PubMed and Scopus databases were searched to identify randomized controlled trials (RCTs) that assessed the impact of computerized decision aids on patient-centered outcomes and SDM in serious illness. Six RCTs were identified and data were extracted on study population, design, and results. Risk of bias was assessed by a modified Cochrane Risk of Bias Tool for Quality Assessment of Randomized Controlled Trials.Six RCTs tested decision tools in varying serious illnesses. Three studies compared different computerized decision aids against each other and a control. All but one study demonstrated improvement in at least one patient-centered outcome. Computerized decision tools may reduce unnecessary treatment in patients with low disease severity in comparison with informational pamphlets. Additionally, electronic health record (EHR) portals may provide the opportunity to manage care from the home for individuals affected by illness. The quality of decision aids is of great importance. Furthermore, satisfaction with the use of tools is associated with increased patient satisfaction and reduced decisional conflict. Finally, patients may benefit from computerized decision tools without the need for increased physician involvement.Most computerized decision aids improved at least one patient-centered outcome. All RCTs identified were at a High Risk of Bias or Unclear Risk of Bias. Effort should be made to improve the quality of RCTs testing SDM aids in serious illness.

Project description:BackgroundIdentifying and managing serious spinal pathology (SSP) such as cauda equina syndrome or spinal infection in patients presenting with low back pain is challenging. Traditional red flag questioning is increasingly criticized, and previous studies show that many clinicians lack confidence in managing patients presenting with red flags. Improving decision-making and reducing the variability of care for these patients is a key priority for clinicians and researchers.ObjectiveWe aimed to improve SSP identification by constructing and validating a decision support tool using a Bayesian network (BN), which is an artificial intelligence technique that combines current evidence and expert knowledge.MethodsA modified RAND appropriateness procedure was undertaken with 16 experts over 3 rounds, designed to elicit the variables, structure, and conditional probabilities necessary to build a causal BN. The BN predicts the likelihood of a patient with a particular presentation having an SSP. The second part of this study used an established framework to direct a 4-part validation that included comparison of the BN with consensus statements, practice guidelines, and recent research. Clinical cases were entered into the model and the results were compared with clinical judgment from spinal experts who were not involved in the elicitation. Receiver operating characteristic curves were plotted and area under the curve were calculated for accuracy statistics.ResultsThe RAND appropriateness procedure elicited a model including 38 variables in 3 domains: risk factors (10 variables), signs and symptoms (17 variables), and judgment factors (11 variables). Clear consensus was found in the risk factors and signs and symptoms for SSP conditions. The 4-part BN validation demonstrated good performance overall and identified areas for further development. Comparison with available clinical literature showed good overall agreement but suggested certain improvements required to, for example, 2 of the 11 judgment factors. Case analysis showed that cauda equina syndrome, space-occupying lesion/cancer, and inflammatory condition identification performed well across the validation domains. Fracture identification performed less well, but the reasons for the erroneous results are well understood. A review of the content by independent spinal experts backed up the issues with the fracture node, but the BN was otherwise deemed acceptable.ConclusionsThe RAND appropriateness procedure and validation framework were successfully implemented to develop the BN for SSP. In comparison with other expert-elicited BN studies, this work goes a step further in validating the output before attempting implementation. Using a framework for model validation, the BN showed encouraging validity and has provided avenues for further developing the outputs that demonstrated poor accuracy. This study provides the vital first step of improving our ability to predict outcomes in low back pain by first considering the problem of SSP.International registered report identifier (irrid)RR2-10.2196/21804.

Project description:ObjectiveTo determine whether a behavioral intervention delivered through a video game can improve the appropriateness of trauma triage decisions in the emergency department of non-trauma centers.DesignRandomized clinical trial.SettingOnline intervention in national sample of emergency medicine physicians who make triage decisions at US hospitals.Participants368 emergency medicine physicians primarily working at non-trauma centers. A random sample (n=200) of those with primary outcome data was reassessed at six months.InterventionsPhysicians were randomized in a 1:1 ratio to one hour of exposure to an adventure video game (Night Shift) or apps based on traditional didactic education (myATLS and Trauma Life Support MCQ Review), both on iPads. Night Shift was developed to recalibrate the process of using pattern recognition to recognize moderate-severe injuries (representativeness heuristics) through the use of stories to promote behavior change (narrative engagement). Physicians were randomized with a 2×2 factorial design to intervention (game v traditional education apps) and then to the experimental condition under which they completed the outcome assessment tool (low v high cognitive load). Blinding could not be maintained after allocation but group assignment was masked during the analysis phase.Main outcome measuresOutcomes of a virtual simulation that included 10 cases; in four of these the patients had severe injuries. Participants completed the simulation within four weeks of their intervention. Decisions to admit, discharge, or transfer were measured. The proportion of patients under-triaged (patients with severe injuries not transferred to a trauma center) was calculated then (primary outcome) and again six months later, with a different set of cases (primary outcome of follow-up study). The secondary outcome was effect of cognitive load on under-triage.Results149 (81%) physicians in the game arm and 148 (80%) in the traditional education arm completed the trial. Of these, 64/100 (64%) and 58/100 (58%), respectively, completed reassessment at six months. The mean age was 40 (SD 8.9), 283 (96%) were trained in emergency medicine, and 207 (70%) were ATLS (advanced trauma life support) certified. Physicians exposed to the game under-triaged fewer severely injured patients than those exposed to didactic education (316/596 (0.53) v 377/592 (0.64), estimated difference 0.11, 95% confidence interval 0.05 to 0.16; P<0.001). Cognitive load did not influence under-triage (161/308 (0.53) v 155/288 (0.54) in the game arm; 197/300 (0.66) v 180/292 (0.62) in the traditional educational apps arm; P=0.66). At six months, physicians exposed to the game remained less likely to under-triage patients (146/256 (0.57) v 172/232 (0.74), estimated difference 0.17, 0.09 to 0.25; P<0.001). No physician reported side effects. The sample might not reflect all emergency medicine physicians, and a small set of cases was used to assess performance.ConclusionsCompared with apps based on traditional didactic education, exposure of physicians to a theoretically grounded video game improved triage decision making in a validated virtual simulation. Though the observed effect was large, the wide confidence intervals include the possibility of a small benefit, and the real world efficacy of this intervention remains uncertain.Trial registrationclinicaltrials.gov; NCT02857348 (initial study)/NCT03138304 (follow-up).

Project description:RationalePulmonologists frequently encounter indeterminate pulmonary nodules in practice, but it is unclear what clinical factors they rely on to guide the diagnostic evaluation.ObjectivesTo assess the current approach to the management of indeterminate pulmonary nodules and to determine the extent to which the addition of a hypothetical diagnostic blood test will influence clinical decision making.MethodsSelected pulmonologists practicing in the United States were invited to participate in a conjoint exercise based on 20 randomly generated cases of varying age, smoking history, and nodule size. Some cases included the result of a hypothetical blood test. Each respondent chose from among three diagnostic options for a patient: noninvasive monitoring (i.e., serial CT or positron emission tomography scan), a minor procedure (i.e., biopsy or bronchoscopy), or a major procedure (i.e., video-assisted thorascopic surgery or thoracotomy). Multivariate logistic regression was used to assess the impact of the three risk factors and the diagnostic blood test on decision making.Measurements and main resultsFour hundred nineteen physicians participated (response rate, 10%). One hundred fifty-three physician surveys met predetermined criteria and were analyzed (4% of all invitees). A diagnostic procedure was recommended for 23% of 6-mm nodules, versus 54, 66, 77, and 84% of nodules 10, 14, 18, and 22 mm, respectively (P?<?0.001). Older age limited recommendations for invasive testing: 54% of 80-year-olds versus 61, 64, 63, and 61% of patients 71, 62, 53, and 44 years of age, respectively (P?<?0.001). In multivariate analyses, nodule size, smoking history, age, and the blood test each influenced decision making (P?<?0.001).ConclusionsThe pulmonologists who participated in this survey were more likely to proceed with invasive testing, instead of observation or additional imaging, as the size of the nodule increased. The use of a hypothetical blood test resulted in significant alterations in the decision to pursue invasive testing.

Dataset Information

Assessing the validity of using serious game technology to analyze physician decision making.

Background

Methods

Findings

Conclusions

Publications

Assessing the validity of using serious game technology to analyze physician decision making.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets