Dataset Information

A Multifaceted Organizational Physician Assessment Program: Validity Evidence and Implications for the Use of Performance Data.

ABSTRACT:

Objective

To provide validity evidence for a multifaceted organizational program for assessing physician performance and evaluate the practical and psychometric consequences of 2 approaches to scoring (mean vs top box scores).

Participants and methods

Participants included physicians with a predominantly outpatient practice in general internal medicine (n=95), neurology (n=99), and psychiatry (n=39) at Mayo Clinic from January 1, 2013, through December 31, 2014. Study measures included hire year, patient complaint and compliment rates, note-signing timeliness, cost per episode of care, and Likert-scaled surveys from patients, learners, and colleagues (scored using mean ratings and top box percentages).

Results

Physicians had a mean ± SD of 0.32±1.78 complaints and 0.12±0.76 compliments per 100 outpatient visits. Most notes were signed on time (mean ± SD, 96%±6.6%). Mean ± SD cost was 0.56±0.59 SDs above the institutional average. Mean ± SD scores were 3.77±0.25 on 4-point and 4.06±0.31 to 4.94±0.08 on 5-point Likert-scaled surveys. Mean ± SD top box scores ranged from 18.6%±16.8% to 90.7%±10.5%. Learner survey scores were positively associated with patient survey scores (r=0.26; P=.003) and negatively associated with years in practice (r=-0.20; P=.02).

Conclusion

This study provides validity evidence for 7 assessments commonly used by medical centers to measure physician performance and reports that top box scores amplify differences among high-performing physicians. These findings inform the most appropriate uses of physician performance data and provide practical guidance to organizations seeking to implement similar assessment programs or use existing performance data in more meaningful ways.

SUBMITTER: Leep Hunderfund AN

PROVIDER: S-EPMC6135024 | biostudies-literature | 2017 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A Multifaceted Organizational Physician Assessment Program: Validity Evidence and Implications for the Use of Performance Data.

Leep Hunderfund Andrea N AN Park Yoon Soo YS Hafferty Frederic W FW Nowicki Kelly M KM Altchuler Steven I SI Reed Darcy A DA

Mayo Clinic proceedings. Innovations, quality & outcomes 20170725 2

<h4>Objective</h4>To provide validity evidence for a multifaceted organizational program for assessing physician performance and evaluate the practical and psychometric consequences of 2 approaches to scoring (mean vs top box scores).<h4>Participants and methods</h4>Participants included physicians with a predominantly outpatient practice in general internal medicine (n=95), neurology (n=99), and psychiatry (n=39) at Mayo Clinic from January 1, 2013, through December 31, 2014. Study measures inc ...[more]

PMID: 30225409

Similar Datasets

Project description:ObjectiveTo examine the extent to which medical group and market factors are related to individual primary care physician (PCP) performance on patient experience measures.Data sourcesThis study employs Clinician and Group CAHPS survey data (n=105,663) from 2,099 adult PCPs belonging to 34 diverse medical groups across California. Medical group directors were interviewed to assess the magnitude and nature of financial incentives directed at individual physicians and the adoption of patient experience improvement strategies. Primary care services area (PCSA) data were used to characterize the market environment of physician practices.Study designWe used multilevel models to estimate the relationship between medical group and market factors and physician performance on each Clinician and Group CAHPS measure. Models statistically controlled for respondent characteristics and accounted for the clustering of respondents within physicians, physicians within medical groups, and medical groups within PCSAs using random effects.Principal findingsCompared with physicians belonging to independent practice associations, physicians belonging to integrated medical groups had better performance on the communication ( p=.007) and care coordination ( p=.03) measures. Physicians belonging to medical groups with greater numbers of PCPs had better performance on all measures. The use of patient experience improvement strategies was not associated with performance. Greater emphasis on productivity and efficiency criteria in individual physician financial incentive formulae was associated with worse access to care ( p=.04). Physicians located in PCSAs with higher area-level deprivation had worse performance on the access to care ( p=.04) and care coordination ( p<.001) measures.ConclusionsPhysicians from integrated medical groups and groups with greater numbers of PCPs performed better on several patient experience measures, suggesting that organized care processes adopted by these groups may enhance patients' experiences. Physicians practicing in markets with high concentrations of vulnerable populations may be disadvantaged by constraints that affect performance. Future studies should clarify the extent to which performance deficits associated with area-level deprivation are modifiable.

Project description:ImportanceDiagnostic acumen is a fundamental skill in the practice of medicine. Scalable, practical, and objective tools to assess diagnostic performance are lacking.ObjectiveTo validate a new method of assessing diagnostic performance that uses automated techniques to assess physicians' diagnostic performance on brief, open-ended case simulations.Design, setting, and participantsRetrospective cohort study of 11 023 unique attempts to solve case simulations on an online software platform, The Human Diagnosis Project (Human Dx). A total of 1738 practicing physicians, residents (internal medicine, family medicine, and emergency medicine), and medical students throughout the United States voluntarily used Human Dx software between January 21, 2016, and January 15, 2017.Main outcomes and measuresInternal structure validity was assessed by 3 measures of diagnostic performance: accuracy, efficiency, and a combined score (Diagnostic Acumen Precision Performance [DAPP]). These were each analyzed by level of training. Association with other variables' validity evidence was evaluated by correlating diagnostic performance and affiliation with an institution ranked in the top 25 medical schools by US News and World Report.ResultsData were analyzed for 239 attending physicians, 926 resident physicians, 347 intern physicians, and 226 medical students. Attending physicians had higher mean accuracy scores than medical students (difference, 8.1; 95% CI, 4.2-12.0; P < .001), as did residents (difference, 8.0; 95% CI, 4.8-11.2; P < .001) and interns (difference, 5.9; 95% CI, 2.3-9.6; P < .001). Attending physicians had higher mean efficiency compared with residents (difference, 4.8; 95% CI, 1.8-7.8; P < .001), interns (difference, 5.0; 95% CI, 1.5-8.4; P = .001), and medical students (difference, 5.4; 95% CI, 1.4-9.3; P = .003). Attending physicians also had significantly higher mean DAPP scores than residents (difference, 2.6; 95% CI, 0.0-5.2; P = .05), interns (difference, 3.6; 95% CI, 0.6-6.6; P = .01), and medical students (difference, 6.7; 95% CI, 3.3-10.2; P < .001). Attending physicians affiliated with a US News and World Report-ranked institution had higher mean DAPP scores compared with nonaffiliated attending physicians (80 [95% CI, 77-83] vs 72 [95% CI, 70-74], respectively; P < .001). Resident physicians affiliated with an institution ranked in the top 25 medical schools by US News and World Report also had higher mean DAPP scores compared with nonaffiliated peers (75 [95% CI, 73-77] vs 71 [95% CI, 69-72], respectively; P < .001).Conclusions and relevanceThe data suggest that diagnostic performance is higher in those with more training and that DAPP scores may be a valid measure to appraise diagnostic performance. This diagnostic assessment tool allows individuals to receive immediate feedback on performance through an openly accessible online platform.

Dataset Information

A Multifaceted Organizational Physician Assessment Program: Validity Evidence and Implications for the Use of Performance Data.

Objective

Participants and methods

Results

Conclusion

Publications

A Multifaceted Organizational Physician Assessment Program: Validity Evidence and Implications for the Use of Performance Data.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets