Dataset Information

Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice.

ABSTRACT: RATIONALE AND OBJECTIVES:Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set. MATERIALS AND METHODS:This institutional review board-approved study randomized 83 radiologists from six Breast Cancer Surveillance Consortium registries to assess one of four test sets of 109 screening mammograms each; 48 radiologists completed a fifth test set of 110 mammograms 2 years later. Test sets differed in number of cancer cases and difficulty of lesion detection. Test set sensitivity and specificity were estimated using woman-level and breast-level recall with cancer status and expert opinion as gold standards. Clinical performance was estimated using women-level recall with cancer status as the gold standard. Spearman rank correlations between test set and clinical performance with 95% confidence intervals (CI) were estimated. RESULTS:For test sets with fewer cancers (N?=?15) that were more difficult to detect, correlations were weak to moderate for sensitivity (woman level?=?0.46, 95% CI?=?0.16, 0.69; breast level?=?0.35, 95% CI?=?0.03, 0.61) and weak for specificity (0.24, 95% CI?=?0.01, 0.45) relative to expert recall. Correlations for test sets with more cancers (N?=?30) were close to 0 and not statistically significant. CONCLUSIONS:Correlations between screening performance on a test set and performance in clinical practice are not strong. Test set performance more accurately reflects performance in clinical practice if cancer prevalence is low and lesions are challenging to detect.

SUBMITTER: Miglioretti DL

PROVIDER: S-EPMC5591765 | biostudies-literature | 2017 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice.

Miglioretti Diana L DL Ichikawa Laura L Smith Robert A RA Buist Diana S M DSM Carney Patricia A PA Geller Berta B Monsees Barbara B Onega Tracy T Rosenberg Robert R Sickles Edward A EA Yankaskas Bonnie C BC Kerlikowske Karla K

Academic radiology 20170524 10

<h4>Rationale and objectives</h4>Evidence is inconsistent about whether radiologists' interpretive performance on a screening mammography test set reflects their performance in clinical practice. This study aimed to estimate the correlation between test set and clinical performance and determine if the correlation is influenced by cancer prevalence or lesion difficulty in the test set.<h4>Materials and methods</h4>This institutional review board-approved study randomized 83 radiologists from six ...[more]

PMID: 28551400

Similar Datasets

Project description:PurposeTo establish the extent to which test set reading can represent actual clinical reporting in screening mammography.Materials and methodsInstitutional ethics approval was granted, and informed consent was obtained from each participating screen reader. The need for informed consent with respect to the use of patient materials was waived. Two hundred mammographic examinations were selected from examinations reported by 10 individual expert screen readers, resulting in 10 reader-specific test sets. Data generated from actual clinical reports were compared with three test set conditions: clinical test set reading with prior images, laboratory test set reading with prior images, and laboratory test set reading without prior images. A further set of five expert screen readers was asked to interpret a common set of images in two identical test set conditions to establish a baseline for intraobserver variability. Confidence scores (from 1 to 4) were assigned to the respective decisions made by readers. Region-of-interest (ROI) figures of merit (FOMs) and side-specific sensitivity and specificity were described for the actual clinical reporting of each reader-specific test set and were compared with those for the three test set conditions. Agreement between pairs of readings was performed by using the Kendall coefficient of concordance.ResultsModerate or acceptable levels of agreement were evident (W = 0.69-0.73, P < .01) when describing group performance between actual clinical reporting and test set conditions that were reasonably close to the established baseline (W = 0.77, P < .01) and were lowest when prior images were excluded. Higher median values for ROI FOMs were demonstrated for the test set conditions than for the actual clinical reporting values; this was possibly linked to changes in sensitivity.ConclusionReasonable levels of agreement between actual clinical reporting and test set conditions can be achieved, although inflated sensitivity may be evident with test set conditions.

Project description:To examine whether U.S. radiologists' interpretive volume affects their screening mammography performance.Annual interpretive volume measures (total, screening, diagnostic, and screening focus [ratio of screening to diagnostic mammograms]) were collected for 120 radiologists in the Breast Cancer Surveillance Consortium (BCSC) who interpreted 783?965 screening mammograms from 2002 to 2006. Volume measures in 1 year were examined by using multivariate logistic regression relative to screening sensitivity, false-positive rates, and cancer detection rate the next year. BCSC registries and the Statistical Coordinating Center received institutional review board approval for active or passive consenting processes and a Federal Certificate of Confidentiality and other protections for participating women, physicians, and facilities. All procedures were compliant with the terms of the Health Insurance Portability and Accountability Act.Mean sensitivity was 85.2% (95% confidence interval [CI]: 83.7%, 86.6%) and was significantly lower for radiologists with a greater screening focus (P = .023) but did not significantly differ by total (P = .47), screening (P = .33), or diagnostic (P = .23) volume. The mean false-positive rate was 9.1% (95% CI: 8.1%, 10.1%), with rates significantly higher for radiologists who had the lowest total (P = .008) and screening (P = .015) volumes. Radiologists with low diagnostic volume (P = .004 and P = .008) and a greater screening focus (P = .003 and P = .002) had significantly lower false-positive and cancer detection rates, respectively. Median invasive tumor size and proportion of cancers detected at early stages did not vary by volume.Increasing minimum interpretive volume requirements in the United States while adding a minimal requirement for diagnostic interpretation could reduce the number of false-positive work-ups without hindering cancer detection. These results provide detailed associations between mammography volumes and performance for policymakers to consider along with workforce, practice organization, and access issues and radiologist experience when reevaluating requirements.

Project description:To develop criteria to identify thresholds for the minimally acceptable performance of physicians interpreting diagnostic mammography studies.In an institutional review board-approved HIPAA-compliant study, an Angoff approach was used to set criteria for identifying minimally acceptable interpretive performance for both workup after abnormal screening examinations and workup of a breast lump. Normative data from the Breast Cancer Surveillance Consortium (BCSC) was used to help the expert radiologist identify the impact of cut points. Simulations, also using data from the BCSC, were used to estimate the expected clinical impact from the recommended performance thresholds.Final cut points for workup of abnormal screening examinations were as follows: sensitivity, less than 80%; specificity, less than 80% or greater than 95%; abnormal interpretation rate, less than 8% or greater than 25%; positive predictive value (PPV) of biopsy recommendation (PPV2), less than 15% or greater than 40%; PPV of biopsy performed (PPV3), less than 20% or greater than 45%; and cancer diagnosis rate, less than 20 per 1000 interpretations. Final cut points for workup of a breast lump were as follows: sensitivity, less than 85%; specificity, less than 83% or greater than 95%; abnormal interpretation rate, less than 10% or greater than 25%; PPV2, less than 25% or greater than 50%; PPV3, less than 30% or greater than 55%; and cancer diagnosis rate, less than 40 per 1000 interpretations. If underperforming physicians moved into the acceptable range after remedial training, the expected result would be (a) diagnosis of an additional 86 cancers per 100,000 women undergoing workup after screening examinations, with a reduction in the number of false-positive examinations by 1067 per 100,000 women undergoing this workup, and (b) diagnosis of an additional 335 cancers per 100,000 women undergoing workup of a breast lump, with a reduction in the number of false-positive examinations by 634 per 100,000 women undergoing this workup.Interpreting physicians who fall outside one or more of the identified cut points should be reviewed in the context of an overall assessment of all their performance measures and their specific practice setting to determine if remedial training is indicated.

Project description:ImportanceDigital mammography (DM) and digital breast tomosynthesis (DBT) are used for routine breast cancer screening. There is minimal evidence on performance outcomes by age, screening round, and breast density in community practice.ObjectiveTo compare DM vs DBT performance by age, baseline vs subsequent screening round, and breast density category.Design, setting, and participantsThis comparative effectiveness study assessed 1 584 079 screening examinations of women aged 40 to 79 years without prior history of breast cancer, mastectomy, or breast augmentation undergoing screening mammography at 46 participating Breast Cancer Surveillance Consortium facilities from January 2010 to April 2018.ExposuresAge, Breast Imaging Reporting and Data System breast density category, screening round, and modality.Main outcomes and measuresAbsolute rates and relative risks (RRs) of screening recall and cancer detection.ResultsOf 1 273 492 DM and 310 587 DBT examinations analyzed, 1 028 891 examinations (65.0%) were of white non-Hispanic women; 399 952 women (25.2%) were younger than 50 years; and 671 136 women (42.4%) had heterogeneously dense or extremely dense breasts. Adjusted differences in DM vs DBT performance were largest on baseline examinations: for example, per 1000 baseline examinations in women ages 50 to 59, recall rates decreased from 241 examinations for DM to 204 examinations for DBT (RR, 0.84; 95% CI, 0.73-0.98), and cancer detection rates increased from 5.9 with DM to 8.8 with DBT (RR, 1.50; 95% CI, 1.10-2.08). On subsequent examinations, women aged 40 to 79 years with heterogeneously dense breasts had improved recall rates and improved cancer detection with DBT. For example, per 1000 examinations in women aged 50 to 59 years, the number of recall examinations decreased from 102 with DM to 93 with DBT (RR, 0.91; 95% CI, 0.84-0.98), and cancer detection increased from 3.7 with DM to 5.3 with DBT (RR, 1.42; 95% CI, 1.23-1.64). Women aged 50 to 79 years with scattered fibroglandular density also had improved recall and cancer detection rates with DBT. Women aged 40 to 49 years with scattered fibroglandular density and women aged 50 to 79 years with almost entirely fatty breasts benefited from improved recall rates without change in cancer detection rates. No improvements in recall or cancer detection rates were observed in women with extremely dense breasts on subsequent examinations for any age group.Conclusions and relevanceThis study found that improvements in recall and cancer detection rates with DBT were greatest on baseline mammograms. On subsequent screening mammograms, the benefits of DBT varied by age and breast density. Women with extremely dense breasts did not benefit from improved recall or cancer detection with DBT on subsequent screening rounds.

Project description:Computer-aided detection identifies suspicious findings on mammograms to assist radiologists. Since the Food and Drug Administration approved the technology in 1998, it has been disseminated into practice, but its effect on the accuracy of interpretation is unclear.We determined the association between the use of computer-aided detection at mammography facilities and the performance of screening mammography from 1998 through 2002 at 43 facilities in three states. We had complete data for 222,135 women (a total of 429,345 mammograms), including 2351 women who received a diagnosis of breast cancer within 1 year after screening. We calculated the specificity, sensitivity, and positive predictive value of screening mammography with and without computer-aided detection, as well as the rates of biopsy and breast-cancer detection and the overall accuracy, measured as the area under the receiver-operating-characteristic (ROC) curve.Seven facilities (16%) implemented computer-aided detection during the study period. Diagnostic specificity decreased from 90.2% before implementation to 87.2% after implementation (P<0.001), the positive predictive value decreased from 4.1% to 3.2% (P=0.01), and the rate of biopsy increased by 19.7% (P<0.001). The increase in sensitivity from 80.4% before implementation of computer-aided detection to 84.0% after implementation was not significant (P=0.32). The change in the cancer-detection rate (including invasive breast cancers and ductal carcinomas in situ) was not significant (4.15 cases per 1000 screening mammograms before implementation and 4.20 cases after implementation, P=0.90). Analyses of data from all 43 facilities showed that the use of computer-aided detection was associated with significantly lower overall accuracy than was nonuse (area under the ROC curve, 0.871 vs. 0.919; P=0.005).The use of computer-aided detection is associated with reduced accuracy of interpretation of screening mammograms. The increased rate of biopsy with the use of computer-aided detection is not clearly associated with improved detection of invasive breast cancer.

Dataset Information

Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice.

Publications

Correlation Between Screening Mammography Interpretive Performance on a Test Set and Performance in Clinical Practice.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets