Dataset Information

Assessment of a Simulated Case-Based Measurement of Physician Diagnostic Performance.

ABSTRACT:

Importance

Diagnostic acumen is a fundamental skill in the practice of medicine. Scalable, practical, and objective tools to assess diagnostic performance are lacking.

Objective

To validate a new method of assessing diagnostic performance that uses automated techniques to assess physicians' diagnostic performance on brief, open-ended case simulations.

Design, setting, and participants

Retrospective cohort study of 11?023 unique attempts to solve case simulations on an online software platform, The Human Diagnosis Project (Human Dx). A total of 1738 practicing physicians, residents (internal medicine, family medicine, and emergency medicine), and medical students throughout the United States voluntarily used Human Dx software between January 21, 2016, and January 15, 2017.

Main outcomes and measures

Internal structure validity was assessed by 3 measures of diagnostic performance: accuracy, efficiency, and a combined score (Diagnostic Acumen Precision Performance [DAPP]). These were each analyzed by level of training. Association with other variables' validity evidence was evaluated by correlating diagnostic performance and affiliation with an institution ranked in the top 25 medical schools by US News and World Report.

Results

Data were analyzed for 239 attending physicians, 926 resident physicians, 347 intern physicians, and 226 medical students. Attending physicians had higher mean accuracy scores than medical students (difference, 8.1; 95% CI, 4.2-12.0; P?Conclusions and relevanceThe data suggest that diagnostic performance is higher in those with more training and that DAPP scores may be a valid measure to appraise diagnostic performance. This diagnostic assessment tool allows individuals to receive immediate feedback on performance through an openly accessible online platform.

SUBMITTER: Chatterjee S

PROVIDER: S-EPMC6484555 | biostudies-literature | 2019 Jan

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Assessment of a Simulated Case-Based Measurement of Physician Diagnostic Performance.

Chatterjee Souvik S Desai Sanjay S Manesh Reza R Sun Junfeng J Nundy Shantanu S Wright Scott M SM

JAMA network open 20190104 1

<h4>Importance</h4>Diagnostic acumen is a fundamental skill in the practice of medicine. Scalable, practical, and objective tools to assess diagnostic performance are lacking.<h4>Objective</h4>To validate a new method of assessing diagnostic performance that uses automated techniques to assess physicians' diagnostic performance on brief, open-ended case simulations.<h4>Design, setting, and participants</h4>Retrospective cohort study of 11 023 unique attempts to solve case simulations on an onlin ...[more]

PMID: 30646211

Similar Datasets

Project description:Importance:Professional guidelines have identified key communication skills for shared decision-making for critically ill patients, but it is unclear how intensivists interpret and implement them. Objective:To compare the self-evaluations of intensivists reviewing transcripts of their own simulated intensive care unit family meetings with the evaluations of trained expert colleagues. Design, Setting, and Participants:A posttrial web-based survey of intensivists was conducted between January and March 2019. Intensivists reviewed transcripts of simulated intensive care unit family meetings in which they participated in a previous trial from October 2016 to November 2017. In the follow-up survey, participants identified if and how they performed key elements of shared decision-making for an intensive care unit patient at high risk of death. Transcript texts that intensivists self-identified as examples of key communication skills recommended by their professional society's policy on shared decision-making were categorized. Main Outcomes and Measures:Comparison of the evaluations of 2 blinded nonparticipant intensivist colleagues with the self-reported responses of the intensivists. Results:Of 116 eligible intensivists, 76 (66%) completed the follow-up survey (mean [SD] respondent age was 43.1 [8.1] years; 72% were male). Sixty-one of 76 intensivists reported conveying prognosis; however, blinded colleagues who reviewed the deidentified transcripts were less likely to report that prognosis had been conveyed than intensivists reviewing their own transcripts (42 of 61; odds ratio, 0.10; 95% CI, 0.01-0.44; P < .001). When reviewing their own transcript, intensivists reported presenting many choices, with the most common choice being code status. They also provided a variety of recommendations, with the most common being to continue the current treatment plan. Thirty-three participants (43%) reported that they offered care focused on comfort, but blinded colleagues rated only 1 (4%) as explaining this option in a clear manner. Conclusions and Relevance:In this study, guidelines for shared decision-making and end of life care were interpreted by intensivists in disparate ways. In the absence of training or personalized feedback, self-assessment of communication skills may not be interpreted consistently.

Project description:Background:Bone age can reflect the true growth and development status of a child; thus, it plays a critical role in evaluating growth and endocrine disorders. This study established and validated an optimized Tanner-Whitehouse 3 artificial intelligence (TW3-AI) bone age assessment (BAA) system based on a convolutional neural network (CNN). Methods:A data set of 9,059 clinical radiographs of the left hand was obtained from the picture archives and communication systems (PACS) between January 2012 and December 2016. Among these, 8,005/9,059 (88%) samples were treated as the training set for model implementation, 804/9,059 (9%) samples as the validation set for parameters optimization, and the remaining 250/9,059 (3%) samples were used to verify the accuracy and reliability of the model compared to that of 4 experienced endocrinologists and 2 experienced radiologists. The overall variation of TW3-metacarpophalangeal, radius, ulna and short bones (RUS) and TW3-Carpal bone score, as well as each bone (13 RUS + 7 Carpal) between reviewers and the AI, were compared by Bland-Altman (BA) chart and Kappa test, respectively. Furthermore, the time consumption between the model and reviewers was also compared. Results:The performance of TW3-AI model was highly consistent with the reviewers' overall estimation, and the root mean square (RMS) was 0.50 years. The accuracy of the BAA of the TW3-AI model was better than the estimate of the reviewers. Further analysis revealed that human interpretations of the male capitate, hamate, the first distal and fifth middle phalanx and female capitate, the trapezoid, and the third and fifth middle phalanx, were most inconsistent. The average image processing time was 1.5±0.2 s in the TW3-AI model, which was significantly shorter than manual interpretation. Conclusions:The diagnostic performance of CNN-based TW3 BAA was accurate and timesaving, and possesses better stability compared to diagnostics made by experienced experts.

Project description:BackgroundPhysician review of a verbal autopsy (VA) and completion of a death certificate remains the most widely used approach for VA analysis. This study provides new evidence about the performance of physician-certified verbal autopsy (PCVA) using defined clinical diagnostic criteria as a gold standard for a multisite sample of 12,542 VAs. The study was also designed to analyze issues related to PCVA, such as the impact of a second physician reader on the cause of death assigned, the variation in performance with and without household recall of health care experience (HCE), and the importance of local information for physicians reading VAs.MethodsThe certification was performed by 24 physicians. The assignment of VA was random and blinded. Each VA was certified by one physician. Half of the VAs were reviewed by a different physician with household recall of health care experience included. The completed death certificate was processed for automated ICD-10 coding of the underlying cause of death. PCVA was compared to gold standard cause of death assignment based on strictly defined clinical diagnostic criteria that are part of the Population Health Metrics Research Consortium (PHMRC) gold standard verbal autopsy study.ResultsFor individual cause assignment, the overall chance-corrected concordance for PCVA against the gold standard cause of death is less than 50%, with substantial variability by cause and physician. Physicians assign the correct cause around 30% of the time without HCE, and addition of HCE improves performance in adults to 45% and slightly higher in children to 48%. Physicians estimate cause-specific mortality fractions (CSMFs) with considerable error for adults, children, and neonates. Only for neonates for a cause list of six causes with HCE is accuracy above 0.7. In all three age groups, CSMF accuracy improves when household recall of health care experience is available.ConclusionsResults show that physician coding for cause of death assignment may not be as robust as previously thought. The time and cost required to initially collect the verbal autopsies must be considered in addition to the analysis, as well as the impact of diverting physicians from servicing immediate health needs in a population to review VAs. All of these considerations highlight the importance and urgency of developing better methods to more reliably analyze past and future verbal autopsies to obtain the highest quality mortality data from populations without reliable death certification.

Project description:Background and objectivesPrecise BP measurement to exclude hypertension is critical in evaluating potential living kidney donors. Ambulatory BP monitoring is considered the gold standard method for diagnosing hypertension, but it is cumbersome to perform. We sought to determine whether lower BP cutoffs using office and automated BP would reduce the rate of missed hypertension in potential living donors.Design, setting, participants, & measurementsWe measured BP in 578 prospective donors using three modalities: (1) single office BP, (2) office automated BP (average of five consecutive automated readings separated by 1 minute), and (3) ambulatory BP. Daytime ambulatory BP was considered the gold standard for diagnosing hypertension. We assessed both the Seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure (JNC-7) and the American College of Cardiology/American Heart Association (ACC/AHA) definitions of hypertension in the cohort. Empirical thresholds of office BP and automated BP for the detection of ambulatory BP-diagnosed hypertension were derived using Youden index, which maximizes the sum of sensitivity and specificity and gives equal weight to false positive and false negative values.ResultsHypertension was diagnosed in 90 (16%) prospective donors by JNC-7 criteria and 198 (34%) prospective donors by ACC/AHA criteria. Masked hypertension was found in 3% of the total cohort by JNC-7 using the combination of office or automated BP, and it was seen in 24% by ACC/AHA guidelines. Using Youden index, cutoffs were derived for both office and automated BP using JNC-7 (<123/82 and <120/78 mm Hg) and ACC/AHA (<119/79 and <116/76 mm Hg) definitions. Using these lower cutoffs, the sensitivity for detecting hypertension improved from 79% to 87% for JNC-7 and from 32% to 87% by ACC/AHA definition, with negative predictive values of 95% and 87%, respectively. Missed (masked) hypertension was reduced to 2% and 4% of the entire cohort by JNC-7and ACC/AHA, respectively.ConclusionsThe prevalence of hypertension was higher in living donor candidates using ACC/AHA compared JNC-7 definitions. Lower BP cutoffs in the clinic improved sensitivity and led to a low overall prevalence of missed hypertension in prospective living kidney donors.

Project description:ObjectivesPediatric appendicitis remains a challenging diagnosis in the emergency department (ED). Available risk prediction algorithms may contribute to excessive ED imaging studies. Incorporation of physician gestalt assessment could help refine predictive tools and improve diagnostic imaging decisions.MethodsThis study was a subanalysis of a parent study that prospectively enrolled patients ages 5 to 20.9 years with a chief complaint of abdominal pain presenting to 11 community EDs within an integrated delivery system between October 1, 2016, and September 30, 2018. Prior to diagnostic imaging, attending emergency physicians enrolled patients with ≤5 days of right-sided or diffuse abdominal pain using a Web-based application embedded in the electronic health record. Predicted risk (gestalt) of acute appendicitis was prospectively entered using a sliding scale from 1% to 100%. As a planned secondary analysis, we assessed the performance of gestalt via c-statistics of receiver operating characteristic (ROC) curves; tested associations between gestalt performance and patient, physician, and facility characteristics; and examined clinical characteristics affecting gestalt estimates.ResultsOf 3,426 patients, 334 (9.8%) had confirmed appendicitis. Physician gestalt had excellent ROC curve characteristics (c-statistic = 0.83, 95% confidence interval = 0.81 to 0.85), performing particularly well in the low-risk strata (appendicitis rate = 1.1% in gestalt 1%-10% range, negative predictive value of 98.9% for appendicitis diagnosis). Physicians with ≥5 years since medical school graduation demonstrated improved gestalt performance over those with less experience (p = 0.007). All clinical characteristics tested, except pain <24 hours, were significantly associated with physician gestalt value (p < 0.05).ConclusionPhysician gestalt for acute appendicitis diagnosis performed well, especially in low-risk patients and when employed by experienced physicians.

Dataset Information

Assessment of a Simulated Case-Based Measurement of Physician Diagnostic Performance.

Importance

Objective

Design, setting, and participants

Main outcomes and measures

Results

Publications

Assessment of a Simulated Case-Based Measurement of Physician Diagnostic Performance.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets