Dataset Information

Association of volume of self-directed versus assigned interpretive work with diagnostic performance of radiologists: an observational study.

ABSTRACT:

Objectives

To understand the sources of variability in diagnostic performance among experienced radiologists.

Design

All prostate MRI examinations performed between 2016 and 2018 were retrospectively reviewed.

Setting

University hospital in Japan.

Participants

Data derived from 334 pathology-proven cases (male, mean age: 70 years; range: 35-90 years) that were interpreted by 10 experienced radiologists were subjected to the analysis.

Primary and secondary outcome measures

Diagnostic performance measures of the radiologists were compared with candidate factors, including interpretive volume of prostate MRIs, volume of self-directed and assigned total annual interpretive work, and years of experience. The potential influence of fatigue was also evaluated by examining the effect of the report's issue time.

Results

There were 186 prostate cancer cases. Performance was based on accuracy, sensitivity and specificity (86%, 85% and 84%, respectively). While performance was not correlated with the volume of prostate MRIs, per se (ρ=-0.15, p=0.69; ρ=-0.01, p=0.99; ρ=-0.33, p=0.36) or the total MRIs assigned for each radiologist (p>0.6) or years of experience (p>0.4), all measures were strongly correlated with voluntary work represented by the interpretive volume of abdominal CTs (r=0.79, p<0.01; r=0.80, p<0.01; r=0.64, p=0.048). The performance did not differ based on the issue time of the report (morning, afternoon and evening) (χ²(2)=3.65, p=0.16).

Conclusions

Greater autonomy, represented as enhanced self-directed interpretive work, was most significantly correlated with the performance of prostate MRI interpretation. The lack of a correlation between the performance and assigned volume confirms the complexity of human learning. Together, these findings support the hypothesis that successful promotion of internal drivers could have a pervasive positive impact on improving diagnostic performance.

SUBMITTER: Amemiya S

PROVIDER: S-EPMC6936980 | biostudies-literature |

REPOSITORIES: biostudies-literature

ACCESS DATA

Similar Datasets

Project description:PurposeTo examine radiologists' screening performance in relation to the number of diagnostic work-ups performed after abnormal findings are discovered at screening mammography by the same radiologist or by different radiologists.Materials and methodsIn an institutional review board-approved HIPAA-compliant study, the authors linked 651 671 screening mammograms interpreted from 2002 to 2006 by 96 radiologists in the Breast Cancer Surveillance Consortium to cancer registries (standard of reference) to evaluate the performance of screening mammography (sensitivity, false-positive rate [ FPR false-positive rate ], and cancer detection rate [ CDR cancer detection rate ]). Logistic regression was used to assess the association between the volume of recalled screening mammograms ("own" mammograms, where the radiologist who interpreted the diagnostic image was the same radiologist who had interpreted the screening image, and "any" mammograms, where the radiologist who interpreted the diagnostic image may or may not have been the radiologist who interpreted the screening image) and screening performance and whether the association between total annual volume and performance differed according to the volume of diagnostic work-up.ResultsAnnually, 38% of radiologists performed the diagnostic work-up for 25 or fewer of their own recalled screening mammograms, 24% performed the work-up for 0-50, and 39% performed the work-up for more than 50. For the work-up of recalled screening mammograms from any radiologist, 24% of radiologists performed the work-up for 0-50 mammograms, 32% performed the work-up for 51-125, and 44% performed the work-up for more than 125. With increasing numbers of radiologist work-ups for their own recalled mammograms, the sensitivity (P = .039), FPR false-positive rate (P = .004), and CDR cancer detection rate (P < .001) of screening mammography increased, yielding a stepped increase in women recalled per cancer detected from 17.4 for 25 or fewer mammograms to 24.6 for more than 50 mammograms. Increases in work-ups for any radiologist yielded significant increases in FPR false-positive rate (P = .011) and CDR cancer detection rate (P = .001) and a nonsignificant increase in sensitivity (P = .15). Radiologists with a lower annual volume of any work-ups had consistently lower FPR false-positive rate , sensitivity, and CDR cancer detection rate at all annual interpretive volumes.ConclusionThese findings support the hypothesis that radiologists may improve their screening performance by performing the diagnostic work-up for their own recalled screening mammograms and directly receiving feedback afforded by means of the outcomes associated with their initial decision to recall. Arranging for radiologists to work up a minimum number of their own recalled cases could improve screening performance but would need systems to facilitate this workflow.

Project description:To develop criteria to identify thresholds for the minimally acceptable performance of physicians interpreting diagnostic mammography studies.In an institutional review board-approved HIPAA-compliant study, an Angoff approach was used to set criteria for identifying minimally acceptable interpretive performance for both workup after abnormal screening examinations and workup of a breast lump. Normative data from the Breast Cancer Surveillance Consortium (BCSC) was used to help the expert radiologist identify the impact of cut points. Simulations, also using data from the BCSC, were used to estimate the expected clinical impact from the recommended performance thresholds.Final cut points for workup of abnormal screening examinations were as follows: sensitivity, less than 80%; specificity, less than 80% or greater than 95%; abnormal interpretation rate, less than 8% or greater than 25%; positive predictive value (PPV) of biopsy recommendation (PPV2), less than 15% or greater than 40%; PPV of biopsy performed (PPV3), less than 20% or greater than 45%; and cancer diagnosis rate, less than 20 per 1000 interpretations. Final cut points for workup of a breast lump were as follows: sensitivity, less than 85%; specificity, less than 83% or greater than 95%; abnormal interpretation rate, less than 10% or greater than 25%; PPV2, less than 25% or greater than 50%; PPV3, less than 30% or greater than 55%; and cancer diagnosis rate, less than 40 per 1000 interpretations. If underperforming physicians moved into the acceptable range after remedial training, the expected result would be (a) diagnosis of an additional 86 cancers per 100,000 women undergoing workup after screening examinations, with a reduction in the number of false-positive examinations by 1067 per 100,000 women undergoing this workup, and (b) diagnosis of an additional 335 cancers per 100,000 women undergoing workup of a breast lump, with a reduction in the number of false-positive examinations by 634 per 100,000 women undergoing this workup.Interpreting physicians who fall outside one or more of the identified cut points should be reviewed in the context of an overall assessment of all their performance measures and their specific practice setting to determine if remedial training is indicated.

Project description:To examine whether U.S. radiologists' interpretive volume affects their screening mammography performance.Annual interpretive volume measures (total, screening, diagnostic, and screening focus [ratio of screening to diagnostic mammograms]) were collected for 120 radiologists in the Breast Cancer Surveillance Consortium (BCSC) who interpreted 783?965 screening mammograms from 2002 to 2006. Volume measures in 1 year were examined by using multivariate logistic regression relative to screening sensitivity, false-positive rates, and cancer detection rate the next year. BCSC registries and the Statistical Coordinating Center received institutional review board approval for active or passive consenting processes and a Federal Certificate of Confidentiality and other protections for participating women, physicians, and facilities. All procedures were compliant with the terms of the Health Insurance Portability and Accountability Act.Mean sensitivity was 85.2% (95% confidence interval [CI]: 83.7%, 86.6%) and was significantly lower for radiologists with a greater screening focus (P = .023) but did not significantly differ by total (P = .47), screening (P = .33), or diagnostic (P = .23) volume. The mean false-positive rate was 9.1% (95% CI: 8.1%, 10.1%), with rates significantly higher for radiologists who had the lowest total (P = .008) and screening (P = .015) volumes. Radiologists with low diagnostic volume (P = .004 and P = .008) and a greater screening focus (P = .003 and P = .002) had significantly lower false-positive and cancer detection rates, respectively. Median invasive tumor size and proportion of cancers detected at early stages did not vary by volume.Increasing minimum interpretive volume requirements in the United States while adding a minimal requirement for diagnostic interpretation could reduce the number of false-positive work-ups without hindering cancer detection. These results provide detailed associations between mammography volumes and performance for policymakers to consider along with workforce, practice organization, and access issues and radiologist experience when reevaluating requirements.

Project description:To evaluate the interobserver variability in descriptions of breast masses by dedicated breast imagers and radiology residents and determine how any differences in lesion description affect the performance of a computer-aided diagnosis (CAD) computer classification system.Institutional review board approval was obtained for this HIPAA-compliant study, and the requirement to obtain informed consent was waived. Images of 50 breast lesions were individually interpreted by seven dedicated breast imagers and 10 radiology residents, yielding 850 lesion interpretations. Lesions were described with use of 11 descriptors from the Breast Imaging Reporting and Data System, and interobserver variability was calculated with the Cohen κ statistic. Those 11 features were selected, along with patient age, and merged together by a linear discriminant analysis (LDA) classification model trained by using 1005 previously existing cases. Variability in the recommendations of the computer model for different observers was also calculated with the Cohen κ statistic.A significant difference was observed for six lesion features, and radiology residents had greater interobserver variability in their selection of five of the six features than did dedicated breast imagers. The LDA model accurately classified lesions for both sets of observers (area under the receiver operating characteristic curve = 0.94 for residents and 0.96 for dedicated imagers). Sensitivity was maintained at 100% for residents and improved from 98% to 100% for dedicated breast imagers. For residents, the computer model could potentially improve the specificity from 20% to 40% (P < .01) and the κ value from 0.09 to 0.53 (P < .001). For dedicated breast imagers, the computer model could increase the specificity from 34% to 43% (P = .16) and the κ value from 0.21 to 0.61 (P < .001).Among findings showing a significant difference, there was greater interobserver variability in lesion descriptions among residents; however, an LDA model using data from either dedicated breast imagers or residents yielded a consistently high performance in the differentiation of benign from malignant breast lesions, demonstrating potential for improving specificity and decreasing interobserver variability in biopsy recommendations.

Project description:ImportanceThe primary data sources used to generate and update work relative value units (RVUs) are surveys of small groups of specialists who are asked to estimate the time and intensity needed to perform surgical procedures. Because these surveys are conducted by specialty societies and rely on subjective data, these sources have been challenged as potentially biased.ObjectiveTo assess whether objective work measures are associated with a surgical procedure's assigned work RVUs and whether differences exist by surgical specialty.Design, setting, and participantsThis cross-sectional study obtained data from the 2016 and 2017 participant use files of the American College of Surgeons National Surgical Quality Improvement Program. The 2017 physician fee schedule of the Centers for Medicare & Medicaid Services was a secondary data source. Procedures were included if they had at least 100 patient-level observations over the 2-year period. Data were analyzed from August 29, 2018, to April 2, 2019.Main outcomes and measuresThe dependent variable was a procedure's assigned work RVU. Independent variables of work RVUs were 4 procedure-level work measures (median operative time, median postoperative length of stay, all-cause 30-day readmission rate, and all-cause 30-day reoperation rate) and surgeon specialty (10-level category using general surgery as the reference).ResultsThe data set included 628 unique Current Procedural Terminology (CPT) codes and 726 CPT-specialty combinations from 1 239 991 patient observations. Statistically significant associations were found between each work measure and assigned work RVU, as follows: median operative time (R2 = 0.74; 95% CI, 0.71-0.78), postoperative length of stay (R2 = 0.42; 95% CI, 0.36-0.48), rate of readmission (R2 = 0.18; 95% CI, 0.13-0.23), and rate of reoperation (R2 = 0.15; 95% CI, 0.10-0.20). Including all 4 measures explained 80.2% (95% CI, 77.3%-83.1%) of the variation. Adding the surgical specialty improved the overall fit of the model (likelihood ratio test χ2 = 231.27; P < .001). Cardiac (7.78; 95% CI, 4.25-11.31; P < .001) and neurosurgery (2.46; 95% CI, 1.08-3.83; P < .001) had higher work RVUs compared with general surgery, whereas orthopedics (-1.53; 95% CI, -2.48 to -0.59; P = .002), urology (-1.58; 95% CI, -2.88 to -0.29; P = .02), plastics (-2.70; 95% CI, -4.39 to -1.01; P = .002), and otolaryngology (-3.05; 95% CI, -4.69 to -1.42; P < .001) had lower work RVUs compared with general surgery.Conclusions and relevanceObjective work measures appeared to be associated with assigned work RVUs, predominantly with operative time; registry data can be used to augment and inform the generation and updating processes of the work RVUs.