Dataset Information

Interrater reliability of Brain Tumor Reporting and Data System (BT-RADS) in the follow up of adult primary brain tumors: a single institution experience in Italy.

ABSTRACT: In 2018, a new system was proposed for classifying and reporting post-treatment adult brain tumor on magnetic resonance imaging, named as Brain Tumor Reporting and Data System (BT-RADS), that needs a validation by means of agreement studies. A retrospective study was designed with the aim of identifying contrast-enhanced magnetic resonance imaging (MRI) of adult patients on follow-up for primary brain tumor at Fondazione Policlinico Campus Bio-Medico. Four radiologists (2 radiology residents, 1 general radiologist, 1 neuroradiologist) read and scored each study using the BT-RADS scoring tool, blinded to the MRI original report. Interobserver agreement and Fleiss' k were calculated to assess the level of diagnostic agreement. It was assessed how many times the assignment of different scoring of BT-RADS would have led to a different patient management. The total number of patients included in the study was 23 with 147 MRIs and a total of 588 BT-RADS scores retrospectively evaluated. The two most frequent tumor types were astrocytoma grade 4 (62%) and oligodendroglioma grade 3 (21%). The overall agreement rate for all 4 radiologists was 82% with a Fleiss' k of 0.70. The overall agreement rate between general radiologist and neuroradiologist was 91% with a Fleiss' k of 0.86. The overall agreement rate between 2 radiology residents and neuroradiologist was 80% with a Fleiss' k of 0.66. Astrocytoma grade 3 (k: 0.51) and oligodendroglioma grade 2 (k: 0.32) showed a poor agreement while higher values of agreement were found for astrocytoma grade 4 (k: 0.70), astrocytoma grade 2 (k: 0.78) and oligodendroglioma grade 3 (k: 0.78). All the radiologists agreed on BT-RADS assignment in 70% patients, three radiologists agreed in 17% and two radiologists agree in 13%. In no cases there was a complete disagreement among the readers. In 18% of cases the discrepancy in the estimated BT-RADS would have led to a different follow-up management. BT-RADS can be considered a valid tool for neuroradiologists and radiologists even with little experience in the interpretation of patients' images during follow-up for adult primary brain tumors supporting standardized interpretation, reporting and clinical management.

SUBMITTER: Parillo M

PROVIDER: S-EPMC10644140 | biostudies-literature | 2023 Nov

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Interrater reliability of Brain Tumor Reporting and Data System (BT-RADS) in the follow up of adult primary brain tumors: a single institution experience in Italy.

Parillo Marco M Mallio Carlo Augusto CA Pileri Matteo M Dirawe Diab D Romano Andrea A Bozzao Alessandro A Weinberg Brent B Quattrocchi Carlo Cosimo CC

Quantitative imaging in medicine and surgery 20230114 11

<h4>Background</h4>In 2018, a new system was proposed for classifying and reporting post-treatment adult brain tumor on magnetic resonance imaging, named as Brain Tumor Reporting and Data System (BT-RADS), that needs a validation by means of agreement studies.<h4>Methods</h4>A retrospective study was designed with the aim of identifying contrast-enhanced magnetic resonance imaging (MRI) of adult patients on follow-up for primary brain tumor at Fondazione Policlinico Campus Bio-Medico. Four radio ...[more]

PMID: 37969622

Similar Datasets

Project description:BackgroundInterrater reliability, aka intercoder reliability, is defined as true agreement between raters, aka coders, without chance agreement. It is used across many disciplines including medical and health research to measure the quality of ratings, coding, diagnoses, or other observations and judgements. While numerous indices of interrater reliability are available, experts disagree on which ones are legitimate or more appropriate. Almost all agree that percent agreement (ao), the oldest and the simplest index, is also the most flawed because it fails to estimate and remove chance agreement, which is produced by raters' random rating. The experts, however, disagree on which chance estimators are legitimate or better. The experts also disagree on which of the three factors, rating category, distribution skew, or task difficulty, an index should rely on to estimate chance agreement, or which factors the known indices in fact rely on. The most popular chance-adjusted indices, according to a functionalist view of mathematical statistics, assume that all raters conduct intentional and maximum random rating while typical raters conduct involuntary and reluctant random rating. The mismatches between the assumed and the actual rater behaviors cause the indices to rely on mistaken factors to estimate chance agreement, leading to the numerous paradoxes, abnormalities, and other misbehaviors of the indices identified by prior studies.MethodsWe conducted a 4 × 8 × 3 between-subject controlled experiment with 4 subjects per cell. Each subject was a rating session with 100 pairs of rating by two raters, totaling 384 rating sessions as the experimental subjects. The experiment tested seven best-known indices of interrater reliability against the observed reliabilities and chance agreements. Impacts of the three factors, i.e., rating category, distribution skew, and task difficulty, on the indices were tested.ResultsThe most criticized index, percent agreement (ao), showed as the most accurate predictor of reliability, reporting directional r2 = .84. It was also the third best approximator, overestimating observed reliability by 13 percentage points on average. The three most acclaimed and most popular indices, Scott's π, Cohen's κ and Krippendorff's α, underperformed all other indices, reporting directional r2 = .312 and underestimated reliability by 31.4 ~ 31.8 points. The newest index, Gwet's AC1, emerged as the second-best predictor and the most accurate approximator. Bennett et al's S ranked behind AC1, and Perreault and Leigh's Ir ranked the fourth both for prediction and approximation. The reliance on category and skew and failure to rely on difficulty explain why the six chance-adjusted indices often underperformed ao, which they were created to outperform. The evidence corroborated the notion that the chance-adjusted indices assume intentional and maximum random rating while the raters instead exhibited involuntary and reluctant random rating.ConclusionThe authors call for more empirical studies and especially more controlled experiments to falsify or qualify this study. If the main findings are replicated and the underlying theories supported, new thinking and new indices may be needed. Index designers may need to refrain from assuming intentional and maximum random rating, and instead assume involuntary and reluctant random rating. Accordingly, the new indices may need to rely on task difficulty, rather than distribution skew or rating category, to estimate chance agreement.

Project description:Enzymatic debridement (ED) has become a reliable tool for eschar removal. Although ED application is simple, wound bed evaluation and therapy decision post-intervention are prone to subjectivity and failure. Experience in ED might be the key, but this has not been proven yet. The aim of this study was to assess interrater reliability (IR) in post-intervention wound bed evaluation and therapy decision as well as the impact of experience. In addition, the authors introduce video assessment as a valuable tool for post-ED decision-making and education. A video-based survey was conducted among physicians with various experiences in ED. The survey involved multiple-choice and 5-point Likert scale questions about professional status, experience in ED, confidence in post-ED wound bed evaluation, and therapy decision. Subsequently, videos of 15 mixed pattern to full-thickness burns immediately after removal of the enzyme complex were demonstrated. Participants were asked for evaluation of each burn wound, including bleeding pattern and consequent therapy decision. IR ≥ 80% was considered as a consensus. Responses were stratified according to participants' experience in applying ED (<10, 10-19, 20-49, and ≥50 applications). IR was assessed by chi-square test (raw agreement [RA]; ≥80% was considered as a consensus) and by calculation of Krippendorff's alpha. In addition, expert consensus for therapy decision was compared with the actual clinical course of each shown patient. Last, participants were asked for their opinion on video as an assessment tool for post-ED wound bed evaluation, decision-making, and training. Thirty-one physicians from 11 burn centers participated in the survey. The overall consensus (raw agreement [RA] ≥ 80%) in post-ED wound bed evaluation and therapy decision was achieved in 20 and 40%, respectively. Krippendorff's alpha is given by 0.32 (95% confidence interval: 0.15, 0.49) and 0.31 (95% confidence interval: 0.16, 0.47), respectively. Subgroup analysis revealed that physicians with high experience in ED achieved significantly more consensus in post-intervention wound bed evaluation and therapy decision compared with physicians with moderate experience (60 vs 13.3%; P = .02 and 86.7 vs 33.3%; P = .04, respectively). Video analysis was considered a feasible (90.3%) and beneficial (93.5%) tool for post-intervention wound bed evaluation and therapy decision as well as useful for training purposes (100%). Reliability of wound bed evaluation and therapy decision after ED depends on the experience of the rating physician. Video analysis is deemed to be a valuable tool for ED evaluation, decision-making, and user training.

Project description:BackgroundThe standard therapy for brain metastasis was surgery combined with whole brain radiotherapy (WBRT). The latter is however, associated with important neurocognitive toxicity. To reduce this toxicity, postoperative stereotactic radiosurgery (SRS) is a promising technique. We assessed the efficacy and the tolerance to postoperative Gamma Knife radiosurgery (GK) on the tumor bed after resection of brain metastases.MethodsBetween February 2011 and December 2016, following macroscopic complete surgical resection, 64 patients and 65 surgical cavities were treated by GK in our institution. The indication for adjuvant radiosurgery was a multidisciplinary decision. The main assessment criteria considered in this study were local control, intracranial metastasis-free survival (ICMFS), overall survival and toxicity.ResultsMedian follow-up: 11.1 months. Median time between surgery and radiosurgery: 35 days. Median dose was 20 Gy prescribed to the 50% isodose line, for a median treated volume of 5.6 cc. Four patients (7%) suffered from local recurrence. Local recurrence-free, intracranial recurrence-free and overall survival at 1 year were 97.5%, 57.6% and 62.4% respectively. In total, 23 patients (41%) suffered from intracranial recurrence outside the tumor bed. In univariate analysis: concomitant GK treatment of multiple lesions and the tumor bed was associated with a decrease in ICMFS (HR = 1.16 [1.005-1.34] p = 0.04). In multivariate analysis: a non-lung primary tumor was significantly associated with a decrease in ICMFS (HR = 8.04 [1.82-35.4] p = 0.006). An increase in performance status (PS) and in the initial number of cerebral metastases significantly reduced overall survival (HR = 5.4 [1.11-26.3] p = 0.037, HR = 2.7 [1.004-7.36] p = 0.049, respectively) and One radiation necrosis histologically proven.ConclusionOur study confirmed that postoperative GK after resection of cerebral metastases is an efficient and well-tolerated technique, to treat volumes of all sizes (0.8 to 40 cc). Iterative SRS or salvage WBRT can be performed in cases of intracranial relapse, postponing WBRT with its potential side effects.

Dataset Information

Interrater reliability of Brain Tumor Reporting and Data System (BT-RADS) in the follow up of adult primary brain tumors: a single institution experience in Italy.

Publications

Interrater reliability of Brain Tumor Reporting and Data System (BT-RADS) in the follow up of adult primary brain tumors: a single institution experience in Italy.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets