Dataset Information

Physician cost profiling--reliability and risk of misclassification.

ABSTRACT: Insurance products with incentives for patients to choose physicians classified as offering lower-cost care on the basis of cost-profiling tools are increasingly common. However, no rigorous evaluation has been undertaken to determine whether these tools can accurately distinguish higher-cost physicians from lower-cost physicians.We aggregated claims data for the years 2004 and 2005 from four health plans in Massachusetts. We used commercial software to construct clinically homogeneous episodes of care (e.g., treatment of diabetes, heart attack, or urinary tract infection), assigned each episode to a physician, and created a summary profile of resource use (i.e., cost) for each physician on the basis of all assigned episodes. We estimated the reliability (signal-to-noise ratio) of each physician's cost-profile score on a scale of 0 to 1, with 0 indicating that all differences in physicians' cost profiles are due to a lack of precision in the measure (noise) and 1 indicating that all differences are due to real variation in costs of services (signal). We used the reliability results to estimate the proportion of physicians in each specialty whose cost performance would be classified inaccurately in a two-tiered insurance product in which the physicians with cost profiles in the lowest quartile were labeled as "lower cost."Median reliabilities ranged from 0.05 for vascular surgery to 0.79 for gastroenterology and otolaryngology. Overall, 59% of physicians had cost-profile scores with reliabilities of less than 0.70, a commonly used marker of suboptimal reliability. Using our reliability results, we estimated that 22% of physicians would be misclassified in a two-tiered system.Current methods for profiling physicians with respect to costs of services may produce misleading results.

SUBMITTER: Adams JL

PROVIDER: S-EPMC2878194 | biostudies-literature | 2010 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Physician cost profiling--reliability and risk of misclassification.

Adams John L JL Mehrotra Ateev A Thomas J William JW McGlynn Elizabeth A EA

The New England journal of medicine 20100301 11

<h4>Background</h4>Insurance products with incentives for patients to choose physicians classified as offering lower-cost care on the basis of cost-profiling tools are increasingly common. However, no rigorous evaluation has been undertaken to determine whether these tools can accurately distinguish higher-cost physicians from lower-cost physicians.<h4>Methods</h4>We aggregated claims data for the years 2004 and 2005 from four health plans in Massachusetts. We used commercial software to constru ...[more]

PMID: 20237347

Similar Datasets

Project description:ImportanceNatural language processing tools, such as ChatGPT (generative pretrained transformer, hereafter referred to as chatbot), have the potential to radically enhance the accessibility of medical information for health professionals and patients. Assessing the safety and efficacy of these tools in answering physician-generated questions is critical to determining their suitability in clinical settings, facilitating complex decision-making, and optimizing health care efficiency.ObjectiveTo assess the accuracy and comprehensiveness of chatbot-generated responses to physician-developed medical queries, highlighting the reliability and limitations of artificial intelligence-generated medical information.Design, setting, and participantsThirty-three physicians across 17 specialties generated 284 medical questions that they subjectively classified as easy, medium, or hard with either binary (yes or no) or descriptive answers. The physicians then graded the chatbot-generated answers to these questions for accuracy (6-point Likert scale with 1 being completely incorrect and 6 being completely correct) and completeness (3-point Likert scale, with 1 being incomplete and 3 being complete plus additional context). Scores were summarized with descriptive statistics and compared using the Mann-Whitney U test or the Kruskal-Wallis test. The study (including data analysis) was conducted from January to May 2023.Main outcomes and measuresAccuracy, completeness, and consistency over time and between 2 different versions (GPT-3.5 and GPT-4) of chatbot-generated medical responses.ResultsAcross all questions (n = 284) generated by 33 physicians (31 faculty members and 2 recent graduates from residency or fellowship programs) across 17 specialties, the median accuracy score was 5.5 (IQR, 4.0-6.0) (between almost completely and complete correct) with a mean (SD) score of 4.8 (1.6) (between mostly and almost completely correct). The median completeness score was 3.0 (IQR, 2.0-3.0) (complete and comprehensive) with a mean (SD) score of 2.5 (0.7). For questions rated easy, medium, and hard, the median accuracy scores were 6.0 (IQR, 5.0-6.0), 5.5 (IQR, 5.0-6.0), and 5.0 (IQR, 4.0-6.0), respectively (mean [SD] scores were 5.0 [1.5], 4.7 [1.7], and 4.6 [1.6], respectively; P = .05). Accuracy scores for binary and descriptive questions were similar (median score, 6.0 [IQR, 4.0-6.0] vs 5.0 [IQR, 3.4-6.0]; mean [SD] score, 4.9 [1.6] vs 4.7 [1.6]; P = .07). Of 36 questions with scores of 1.0 to 2.0, 34 were requeried or regraded 8 to 17 days later with substantial improvement (median score 2.0 [IQR, 1.0-3.0] vs 4.0 [IQR, 2.0-5.3]; P < .01). A subset of questions, regardless of initial scores (version 3.5), were regenerated and rescored using version 4 with improvement (mean accuracy [SD] score, 5.2 [1.5] vs 5.7 [0.8]; median score, 6.0 [IQR, 5.0-6.0] for original and 6.0 [IQR, 6.0-6.0] for rescored; P = .002).Conclusions and relevanceIn this cross-sectional study, chatbot generated largely accurate information to diverse medical queries as judged by academic physician specialists with improvement over time, although it had important limitations. Further research and model development are needed to correct inaccuracies and for validation.

Project description:BackgroundPharmaceutical costs are the fastest-growing health-care expense in most developed countries. Higher drug costs have been shown to negatively impact patient outcomes. Studies suggest that doctors have a poor understanding of pharmaceutical costs, but the data are variable and there is no consistent pattern in awareness. We designed this systematic review to investigate doctors' knowledge of the relative and absolute costs of medications and to determine the factors that influence awareness.Methods and findingsOur search strategy included The Cochrane Library, EconoLit, EMBASE, and MEDLINE as well as reference lists and contact with authors who had published two or more articles on the topic or who had published within 10 y of the commencement of our review. Studies were included if: either doctors, trainees (interns or residents), or medical students were surveyed; there were more than ten survey respondents; cost of pharmaceuticals was estimated; results were expressed quantitatively; there was a clear description of how authors defined "accurate estimates"; and there was a description of how the true cost was determined. Two authors reviewed each article for eligibility and extracted data independently. Cost accuracy outcomes were summarized, but data were not combined in meta-analysis because of extensive heterogeneity. Qualitative data related to physicians and drug costs were also extracted. The final analysis included 24 articles. Cost accuracy was low; 31% of estimates were within 20% or 25% of the true cost, and fewer than 50% were accurate by any definition of cost accuracy. Methodological weaknesses were common, and studies of low methodological quality showed better cost awareness. The most important factor influencing the pattern and accuracy of estimation was the true cost of therapy. High-cost drugs were estimated more accurately than inexpensive ones (74% versus 31%, Chi-square p < 0.001). Doctors consistently overestimated the cost of inexpensive products and underestimated the cost of expensive ones (binomial test, 89/101, p < 0.001). When asked, doctors indicated that they want cost information and feel it would improve their prescribing but that it is not accessible.ConclusionsDoctors' ignorance of costs, combined with their tendency to underestimate the price of expensive drugs and overestimate the price of inexpensive ones, demonstrate a lack of appreciation of the large difference in cost between inexpensive and expensive drugs. This discrepancy in turn could have profound implications for overall drug expenditures. Much more focus is required in the education of physicians about costs and the access to cost information. Future research should focus on the accessibility and reliability of medical cost information and whether the provision of this information is used by doctors and makes a difference to physician prescribing. Additionally, future work should strive for higher methodological standards to avoid the biases we found in the current literature, including attention to the method of assessing accuracy that allows larger absolute estimation ranges for expensive drugs.

Project description:BACKGROUND:Patient experience measures are widely used to compare performance at the individual physician level. OBJECTIVE:To assess the impact of unmeasured patient characteristics on visit-level patient experience measures and the sample sizes required to reliably measure patient experience at the primary care physician (PCP) level. DESIGN:Repeated cross-sectional design. SETTING:Academic family medicine practice in California. PARTICIPANTS:One thousand one hundred forty-one adult patients attending 1319 visits with 56 PCPs (including 45 resident and 11 faculty physicians). MEASUREMENTS:Post-visit patient experience surveys including patient measures used for standard adjustment as recommend by the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Consortium and additional patient characteristics used for expanded adjustment (including attitudes toward healthcare, global life satisfaction, patient personality, current symptom bother, and marital status). RESULTS:The amount of variance in patient experience explained doubled with expanded adjustment for patient characteristics compared with standard adjustment (R2 = 20.0% vs. 9.6%, respectively). With expanded adjustment, the amount of variance attributable to the PCP dropped from 6.1% to 3.4% and the required sample size to achieve a reliability of 0.90 in the physician-level patient experience measure increased from 138 to 255 patients per physician. After ranking of the 56 PCPs by average patient experience, 8 were reclassified into or out of the top or bottom quartiles of average experience with expanded as compared to standard adjustment [14.3% (95% CI: 7.0-25.2%)]. CONCLUSIONS:Widely used methods for measuring PCP-level patient experience may not account sufficiently for influential patient characteristics. If methods were adapted to account for these characteristics, patient sample sizes for reliable between-physician comparisons may be too large for most practices to obtain.

Dataset Information

Physician cost profiling--reliability and risk of misclassification.

Publications

Physician cost profiling--reliability and risk of misclassification.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets