Dataset Information

Prognostic value of histopathological DCIS features in a large-scale international interrater reliability study.

ABSTRACT: PURPOSE:For optimal management of ductal carcinoma in situ (DCIS), reproducible histopathological assessment is essential to distinguish low-risk from high-risk DCIS. Therefore, we analyzed interrater reliability of histopathological DCIS features and assessed their associations with subsequent ipsilateral invasive breast cancer (iIBC) risk. METHODS:Using a case-cohort design, reliability was assessed in a population-based, nationwide cohort of 2767 women with screen-detected DCIS diagnosed between 1993 and 2004, treated by breast-conserving surgery with/without radiotherapy (BCS?±?RT) using Krippendorff's alpha (KA) and Gwet's AC2 (GAC2). Thirty-eight raters scored histopathological DCIS features including grade (2-tiered and 3-tiered), growth pattern, mitotic activity, periductal fibrosis, and lymphocytic infiltrate in 342 women. Using majority opinion-based scores for each feature, their association with subsequent iIBC risk was assessed using Cox regression. RESULTS:Interrater reliability of grade using various classifications was fair to moderate, and only substantial for grade 1 versus 2?+?3 when using GAC2 (0.78). Reliability for growth pattern (KA 0.44, GAC2 0.78), calcifications (KA 0.49, GAC2 0.70) and necrosis (KA 0.47, GAC2 0.70) was moderate using KA and substantial using GAC2; for (type of) periductal fibrosis and lymphocytic infiltrate fair to moderate estimates were found and for mitotic activity reliability was substantial using GAC2 (0.70). Only in patients treated with BCS-RT, high mitotic activity was associated with a higher iIBC risk in univariable analysis (Hazard Ratio (HR) 2.53, 95% Confidence Interval (95% CI) 1.05-6.11); grade 3 versus 1?+?2 (HR 2.64, 95% CI 1.35-5.14) and a cribriform/solid versus flat epithelial atypia/clinging/(micro)papillary growth pattern (HR 3.70, 95% CI 1.34-10.23) were independently associated with a higher iIBC risk. CONCLUSIONS:Using majority opinion-based scores, DCIS grade, growth pattern, and mitotic activity are associated with iIBC risk in patients treated with BCS-RT, but interrater variability is substantial. Semi-quantitative grading, incorporating and separately evaluating nuclear pleomorphism, growth pattern, and mitotic activity, may improve the reliability and prognostic value of these features.

SUBMITTER: Groen EJ

PROVIDER: S-EPMC7497690 | biostudies-literature | 2020 Oct

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Prognostic value of histopathological DCIS features in a large-scale international interrater reliability study.

Groen Emma J EJ Hudecek Jan J Mulder Lennart L van Seijen Maartje M Almekinders Mathilde M MM Alexov Stoyan S Kovács Anikó A Ryska Ales A Varga Zsuzsanna Z Andreu Navarro Francisco-Javier FJ Bianchi Simonetta S Vreuls Willem W Balslev Eva E Boot Max V MV Kulka Janina J Chmielik Ewa E Barbé Ellis E de Rooij Mathilda J MJ Vos Winand W Farkas Andrea A Leeuwis-Fedorovich Natalja E NE Regitnig Peter P Westenend Pieter J PJ Kooreman Loes F S LFS Quinn Cecily C Floris Giuseppe G Cserni Gábor G van Diest Paul J PJ Lips Esther H EH Schaapveld Michael M Wesseling Jelle J

Breast cancer research and treatment 20200730 3

<h4>Purpose</h4>For optimal management of ductal carcinoma in situ (DCIS), reproducible histopathological assessment is essential to distinguish low-risk from high-risk DCIS. Therefore, we analyzed interrater reliability of histopathological DCIS features and assessed their associations with subsequent ipsilateral invasive breast cancer (iIBC) risk.<h4>Methods</h4>Using a case-cohort design, reliability was assessed in a population-based, nationwide cohort of 2767 women with screen-detected DCIS ...[more]

PMID: 32734520

Similar Datasets

Project description:BackgroundInterrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.MethodsWe conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.ResultsThe most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC1, and Perreault and Leigh's Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.ConclusionThe authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.

Project description:ObjectiveTo investigate the effects of Image Biomarker Standardisation Initiative (IBSI) compliance, harmonisation of calculation settings and platform version on the statistical reliability of radiomic features and their corresponding ability to predict clinical outcome.MethodsThe statistical reliability of radiomic features was assessed retrospectively in three clinical datasets (patient numbers: 108 head and neck cancer, 37 small-cell lung cancer, 47 non-small-cell lung cancer). Features were calculated using four platforms (PyRadiomics, LIFEx, CERR and IBEX). PyRadiomics, LIFEx and CERR are IBSI-compliant, whereas IBEX is not. The effects of IBSI compliance, user-defined calculation settings and platform version were assessed by calculating intraclass correlation coefficients and confidence intervals. The influence of platform choice on the relationship between radiomic biomarkers and survival was evaluated using univariable cox regression in the largest dataset.ResultsThe reliability of radiomic features calculated by the different software platforms was only excellent (ICC > 0.9) for 4/17 radiomic features when comparing all four platforms. Reliability improved to ICC > 0.9 for 15/17 radiomic features when analysis was restricted to the three IBSI-compliant platforms. Failure to harmonise calculation settings resulted in poor reliability, even across the IBSI-compliant platforms. Software platform version also had a marked effect on feature reliability in CERR and LIFEx. Features identified as having significant relationship to survival varied between platforms, as did the direction of hazard ratios.ConclusionIBSI compliance, user-defined calculation settings and choice of platform version all influence the statistical reliability and corresponding performance of prognostic models in radiomics.Key points• Reliability of radiomic features varies between feature calculation platforms and with choice of software version. • Image Biomarker Standardisation Initiative (IBSI) compliance improves reliability of radiomic features across platforms, but only when calculation settings are harmonised. • IBSI compliance, user-defined calculation settings and choice of platform version collectively affect the prognostic value of features.

Dataset Information

Prognostic value of histopathological DCIS features in a large-scale international interrater reliability study.

Publications

Prognostic value of histopathological DCIS features in a large-scale international interrater reliability study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets