Dataset Information

Improving the reliability of physician "report cards".

ABSTRACT: Performance measures are widely used to profile primary care physicians (PCPs) but their reliability is often limited by small sample sizes. We evaluated the reliability of individual PCP profiles and whether they can be improved by combining measures into composites or by profiling practice groups.We performed a cross-sectional analysis of electronic health record data for patients with diabetes (DM), congestive heart failure (CHF), ischemic vascular disease (IVD), or eligible for preventive care services seen by a PCP within a large, integrated health care system between April 2009 and May 2010. We evaluated performance on 14 measures of DM care, 9 of CHF, 7 of IVD, and 4 of preventive care.There were 51,771 patients observed by 163 physicians in 17 clinics. Few PCPs (0%-60%) could be profiled with 80% reliability using single process or intermediate-outcome measures. Combining measures into single-disease composites improved reliability for DM and preventive care with 74.5% and 76.7% of PCPs having sufficient panel sizes, but composites remained unreliable for CHF and IVD. A total of 85.3% of PCPs could be reliably profiled using a single overall composite. Aggregating PCPs into practice groups (3 to 21 PCPs per group) did not improve reliability in most cases because of little between-group practice variation.Single measures rarely differentiate between individual PCPs or groups of PCPs reliably. Combining measures into single-disease or multidisease composites can improve reliability for some common conditions, but not all. Assessing PCP practice groups within a single health care system, rather than individual PCPs, did not substantially improve reliability.

SUBMITTER: Smith KA

PROVIDER: S-EPMC3669898 | biostudies-literature | 2013 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Improving the reliability of physician "report cards".

Smith Kimberly A KA Sussman Jeremy B JB Bernstein Steven J SJ Hayward Rodney A RA

Medical care 20130301 3

<h4>Background</h4>Performance measures are widely used to profile primary care physicians (PCPs) but their reliability is often limited by small sample sizes. We evaluated the reliability of individual PCP profiles and whether they can be improved by combining measures into composites or by profiling practice groups.<h4>Methods</h4>We performed a cross-sectional analysis of electronic health record data for patients with diabetes (DM), congestive heart failure (CHF), ischemic vascular disease ( ...[more]

PMID: 23295578

Similar Datasets

Project description:BackgroundThe goal of radical prostatectomy is to achieve the optimal balance between complete cancer removal and preserving a patient's urinary and sexual function. Performing a wider excision of peri-prostatic tissue helps achieve negative surgical margins, but can compromise urinary and sexual function. Alternatively, sparing peri-prostatic tissue to maintain functional outcomes may result in an increased risk of cancer recurrence. The objective of this study is to determine the effect of providing surgeons with detailed information about their patient outcomes through a surgical report card.MethodsWe propose a prospective cohort quasi-experimental study. The intervention is the provision of feedback to prostate cancer surgeons via surgical report cards. These report cards will be distributed every 3 months by email and will present surgeons with detailed information, including urinary function, erectile function, and surgical margin outcomes of their patients compared to patients treated by other de-identified surgeons in the study. For the first 12 months of the study, pre-operative, 6-month, and 12-month patient data will be collected but there will be no report cards distributed to surgeons. This will form the pre-feedback cohort. After the pre-feedback cohort has completed accrual, surgeons will receive quarterly report cards. Patients treated after the provision of report cards will comprise the post-feedback cohort. The primary comparison will be post-operative function of the pre-feedback cohort vs. post-feedback cohort. The secondary comparison will be the proportion of patients with positive surgical margins in the two cohorts. Outcomes will be stratified or case-mix adjusted, as appropriate. Assuming a baseline potency of 20% and a baseline continence of 70%, 292 patients will be required for 80% power at an alpha of 5% to detect a 10% improvement in functional outcomes. Assuming 30% of patients may be lost to follow-up, a minimum sample size of 210 patients is required in the pre-feedback cohort and 210 patients in the post-feedback cohort.DiscussionThe findings from this study will have an immediate impact on surgeon self-evaluation and we hypothesize surgical report cards will result in improved overall outcomes of men treated with radical prostatectomy.

Project description:Doctors commencing Foundation Year (FY) training face many stresses and challenges. FY doctors are often the first point of contact for acutely unwell and deteriorating patients. Trust guidelines are used to aid acute medical management. Accessing guidelines is often fraught with barriers. Evidence suggests aide-memoire cards can provide easier access to guidelines and management pathways. We aimed to improve prescribing accuracy and efficiency of FY doctors for acute medical conditions within Gloucestershire trust by improving access to and usability of trust guidelines. Questionnaires were distributed to FY doctors to identify acute medical conditions to include on the emergency prescription cards (EPCs). Two small double-sided cards were created containing bullet pointed trust guidelines for: hyper/hypokalaemia, status epilepticus, diabetic emergencies, arrhythmias, myocardial infarction, acute asthma, pulmonary oedema, anaphylaxis and a ward-round checklist. Feedback was used to improve EPCs prior to distribution. Pre (N=53) and post-intervention (N=46) written questionnaires were completed by FY doctors. These assessed acute clinical management including use of guidance, confidence in management, speed of prescribing and EPC "usability". To assess prescribing accuracy, prescriptions for acute medical conditions were reviewed pre (N=8) and post-intervention (N=12). The EPCs were well received (80% quite/very useful) and found "easy to use" (83%). The introduction of EPCs increased guidance use (pre-intervention 58.8%, post-intervention 71.7%), increased confidence (pre-intervention 79%, post-intervention 89%) and significantly improved prescribing speed (p=0.05). There was a significant correlation with confidence and prescribing speed (p = 0.023). The accuracy of prescribed doses improved (pre-intervention 62.5%, post-intervention 87.5% accurate) as did details regarding route / additional required information (pre-intervention 75%, post-intervention 97.7%). The EPCs support the management of unwell patients, are relevant to the workload of modern doctors practice and may improve patient care. This improvement measure could be applied to other NHS trusts and medical specialties.

Project description:ContextMany of the publicly available health quality report cards are based on administrative data. ICD-9-CM codes in administrative data are not date stamped to distinguish between medical conditions present at the time of hospital admission and complications, which occur after hospital admission. Treating complications as preexisting conditions gives poor-performing hospitals "credit" for their complications and may cause some hospitals that are delivering low-quality care to be misclassified as average- or high-performing hospitals.ObjectiveTo determine whether hospital quality assessment based on administrative data is impacted by the inclusion of condition present at admission (CPAA) modifiers in administrative data as a date stamp indicator.Design, setting, and patientsRetrospective cohort study based on 648,866 inpatient admissions between 1998 and 2000 for coronary artery bypass graft (CABG) surgery, coronary angioplasty (PTCA), carotid endarterectomy (CEA), abdominal aortic aneurysm (AAA) repair, total hip replacement (THR), acute MI (AMI), and stroke using the California State Inpatient Database which includes CPAA modifiers. Hierarchical logistic regression was used to create separate condition-specific risk adjustment models. For each study population, one model was constructed using only secondary diagnoses present at admission based on the CPAA modifier: "date stamp" model. The second model was constructed using all secondary diagnoses, ignoring the information present in the CPAA modifier: the "no date stamp model." Hospital quality was assessed separately using the "date stamp" and the "no date stamp" risk-adjustment models.ResultsForty percent of the CABG hospitals, 33 percent of the PTCA hospitals, 40 percent of the THR hospitals, and 33 percent of the AMI hospitals identified as low-performance hospitals by the "date stamp" models were not classified as low-performance hospitals by the "no date stamp" models. Fifty percent of the CABG hospitals, 33 percent of the PTCA hospitals, 50 percent of the CEA hospitals, and 36 percent of the AMI hospitals identified as low-performance hospitals by the "no date stamp" models were not identified as low-performance hospitals by the "date stamp" models. The inclusion of the CPAA modifier had a minor impact on hospital quality assessment for AAA repair, stroke, and CEA.ConclusionThis study supports the hypothesis that the use of routine administrative data without date stamp information to construct hospital quality report cards may result in the mis-identification of hospital quality outliers. However, the CPAA modifier will need to be further validated before date stamped administrative data can be used as the basis for health quality report cards.

Project description:ImportanceNatural language processing tools, such as ChatGPT (generative pretrained transformer, hereafter referred to as chatbot), have the potential to radically enhance the accessibility of medical information for health professionals and patients. Assessing the safety and efficacy of these tools in answering physician-generated questions is critical to determining their suitability in clinical settings, facilitating complex decision-making, and optimizing health care efficiency.ObjectiveTo assess the accuracy and comprehensiveness of chatbot-generated responses to physician-developed medical queries, highlighting the reliability and limitations of artificial intelligence-generated medical information.Design, setting, and participantsThirty-three physicians across 17 specialties generated 284 medical questions that they subjectively classified as easy, medium, or hard with either binary (yes or no) or descriptive answers. The physicians then graded the chatbot-generated answers to these questions for accuracy (6-point Likert scale with 1 being completely incorrect and 6 being completely correct) and completeness (3-point Likert scale, with 1 being incomplete and 3 being complete plus additional context). Scores were summarized with descriptive statistics and compared using the Mann-Whitney U test or the Kruskal-Wallis test. The study (including data analysis) was conducted from January to May 2023.Main outcomes and measuresAccuracy, completeness, and consistency over time and between 2 different versions (GPT-3.5 and GPT-4) of chatbot-generated medical responses.ResultsAcross all questions (n = 284) generated by 33 physicians (31 faculty members and 2 recent graduates from residency or fellowship programs) across 17 specialties, the median accuracy score was 5.5 (IQR, 4.0-6.0) (between almost completely and complete correct) with a mean (SD) score of 4.8 (1.6) (between mostly and almost completely correct). The median completeness score was 3.0 (IQR, 2.0-3.0) (complete and comprehensive) with a mean (SD) score of 2.5 (0.7). For questions rated easy, medium, and hard, the median accuracy scores were 6.0 (IQR, 5.0-6.0), 5.5 (IQR, 5.0-6.0), and 5.0 (IQR, 4.0-6.0), respectively (mean [SD] scores were 5.0 [1.5], 4.7 [1.7], and 4.6 [1.6], respectively; P = .05). Accuracy scores for binary and descriptive questions were similar (median score, 6.0 [IQR, 4.0-6.0] vs 5.0 [IQR, 3.4-6.0]; mean [SD] score, 4.9 [1.6] vs 4.7 [1.6]; P = .07). Of 36 questions with scores of 1.0 to 2.0, 34 were requeried or regraded 8 to 17 days later with substantial improvement (median score 2.0 [IQR, 1.0-3.0] vs 4.0 [IQR, 2.0-5.3]; P < .01). A subset of questions, regardless of initial scores (version 3.5), were regenerated and rescored using version 4 with improvement (mean accuracy [SD] score, 5.2 [1.5] vs 5.7 [0.8]; median score, 6.0 [IQR, 5.0-6.0] for original and 6.0 [IQR, 6.0-6.0] for rescored; P = .002).Conclusions and relevanceIn this cross-sectional study, chatbot generated largely accurate information to diverse medical queries as judged by academic physician specialists with improvement over time, although it had important limitations. Further research and model development are needed to correct inaccuracies and for validation.

Project description:BackgroundPublic reporting of provider performance can assist consumers in their choice of providers and stimulate providers to improve quality. Reporting of quality measures is supported by advocates of health care reform across the political spectrum.ObjectiveTo assess the availability, credibility and applicability of existing public reports of hospital and physician quality, with comparisons across geographic areas.ApproachInformation pertaining to 263 public reports in 21 geographic areas was collected through reviews of websites and telephone and in-person interviews, and used to construct indicators of public reporting status. Interview data collected in 14 of these areas were used to assess recent changes in reporting and their implications.ParticipantsInterviewees included staff of state and local associations, health plan representatives and leaders of local health care alliances.ResultsThere were more reports of hospital performance (161) than of physician performance (103) in the study areas. More reports included measures derived from claims data (mean, 7.2 hospital reports and 3.3 physician reports per area) than from medical records data. Typically, reports on physician performance contained measures of chronic illness treatment constructed at the medical group level, with diabetes measures the most common (mean number per non-health plan report, 2.3). Patient experience measures were available in more hospital reports (mean number of reports, 1.2) than physician reports (mean, 0.7). Despite the availability of national hospital reports and reports sponsored by national health plans, from a consumer standpoint the status of public reporting depended greatly on where one lived and health plan membership.ConclusionsCurrent public reports, and especially reports of physician quality of care, have significant limitations from both consumer and provider perspectives. The present approach to reporting is being challenged by the development of new information sources for consumers, and consumer and provider demands for more current information.

Dataset Information

Improving the reliability of physician "report cards".

Publications

Improving the reliability of physician "report cards".

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets