Dataset Information

Intra- and interreader reproducibility of PI-RADSv2: A multireader study.

ABSTRACT: BACKGROUND:The Prostate Imaging Reporting and Data System version 2 (PI-RADSv2) has been in use since 2015; while interreader reproducibility has been studied, there has been a paucity of studies investigating the intrareader reproducibility of PI-RADSv2. PURPOSE:To evaluate both intra- and interreader reproducibility of PI-RADSv2 in the assessment of intraprostatic lesions using multiparametric magnetic resonance imaging (mpMRI). STUDY TYPE:Retrospective. POPULATION/SUBJECTS:In all, 102 consecutive biopsy-naïve patients who underwent prostate MRI and subsequent MR/transrectal ultrasonography (MR/TRUS)-guided biopsy. FIELD STRENGTH/SEQUENCES:Prostate mpMRI at 3T using endorectal with phased array surface coils (TW MRI, DW MRI with ADC maps and b2000 DW MRI, DCE MRI). ASSESSMENT:Previously detected and biopsied lesions were scored by four readers from four different institutions using PI-RADSv2. Readers scored lesions during two readout rounds with a 4-week washout period. STATISTICAL TESTS:Kappa (?) statistics and specific agreement (Po ) were calculated to quantify intra- and interreader reproducibility of PI-RADSv2 scoring. Lesion measurement agreement was calculated using the intraclass correlation coefficient (ICC). RESULTS:Overall intrareader reproducibility was moderate to substantial (??=?0.43-0.67, Po ?=?0.60-0.77), while overall interreader reproducibility was poor to moderate (??=?0.24, Po ?=?46). Readers with more experience showed greater interreader reproducibility than readers with intermediate experience in the whole prostate (P?=?0.026) and peripheral zone (P?=?0.002). Sequence-specific interreader agreement for all readers was similar to the overall PI-RADSv2 score, with ??=?0.24, 0.24, and 0.23 and Po ?=?0.47, 0.44, and 0.54 in T2 -weighted, diffusion-weighted imaging (DWI), and dynamic contrast-enhanced (DCE), respectively. Overall intrareader and interreader ICC for lesion measurement was 0.82 and 0.71, respectively. DATA CONCLUSION:PI-RADSv2 provides moderate intrareader reproducibility, poor interreader reproducibility, and moderate interreader lesion measurement reproducibility. These findings suggest a need for more standardized reader training in prostate MRI. LEVEL OF EVIDENCE:2 Technical Efficacy: Stage 2.

SUBMITTER: Smith CP

PROVIDER: S-EPMC6504619 | biostudies-literature | 2019 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Intra- and interreader reproducibility of PI-RADSv2: A multireader study.

Smith Clayton P CP Harmon Stephanie A SA Barrett Tristan T Bittencourt Leonardo K LK Law Yan Mee YM Shebel Haytham H An Julie Y JY Czarniecki Marcin M Mehralivand Sherif S Coskun Mehmet M Wood Bradford J BJ Pinto Peter A PA Shih Joanna H JH Choyke Peter L PL Turkbey Baris B

Journal of magnetic resonance imaging : JMRI 20181221 6

<h4>Background</h4>The Prostate Imaging Reporting and Data System version 2 (PI-RADSv2) has been in use since 2015; while interreader reproducibility has been studied, there has been a paucity of studies investigating the intrareader reproducibility of PI-RADSv2.<h4>Purpose</h4>To evaluate both intra- and interreader reproducibility of PI-RADSv2 in the assessment of intraprostatic lesions using multiparametric magnetic resonance imaging (mpMRI).<h4>Study type</h4>Retrospective.<h4>Population/sub ...[more]

PMID: 30575184

Similar Datasets

Project description:ObjectiveThe purposes of this study were to assess correlation of apparent diffusion coefficient (ADC) and normalized ADC (ratio of tumor to nontumor tissue) with the Prostate Imaging Reporting and Data System version 2 (PI-RADSv2) and updated International Society of Urological Pathology (ISUP) categories and to determine how to optimally use ADC metrics for objective assistance in categorizing lesions within PI-RADSv2 guidelines.Materials and methodsIn this retrospective study, 100 patients (median age, 62 years; range, 44-75 years; prostate-specific antigen level, 7.18 ng/mL; range, 1.70-84.56 ng/mL) underwent 3-T multiparametric MRI of the prostate with an endorectal coil. Mean ADC was extracted from ROIs based on subsequent prostatectomy specimens. Histopathologic analysis revealed 172 lesions (113 peripheral, 59 transition zone). Two radiologists blinded to histopathologic outcome assigned PI-RADSv2 categories. Kendall tau was used to correlate ADC metrics with PI-RADSv2 and ISUP categories. ROC curves were used to assess the utility of ADC metrics in differentiating each reader's PI-RADSv2 DWI category 4 or 5 assessment in the whole prostate and by zone.ResultsADC metrics negatively correlated with ISUP category in the whole prostate (ADC, τ = -0.21, p = 0.0002; normalized ADC, τ = -0.21, p = 0.0001). Moderate negative correlation was found in expert PI-RADSv2 DWI categories (ADC, τ = -0.34; normalized ADC, τ = -0.31; each p < 0.0001) maintained across zones. In the whole prostate, AUCs of ADC and normalized ADC were 87% and 82% for predicting expert PI-RADSv2 DWI category 4 or 5. A derived optimal cutoff ADC less than 1061 and normalized ADC less than 0.65 achieved positive predictive values of 83% and 84% for correct classification of PI-RADSv2 DWI category 4 or 5 by an expert reader. Consistent relations and predictive values were found by an independent novice reader.ConclusionADC and normalized ADC inversely correlate with PI-RADSv2 and ISUP categories and can serve as quantitative metrics to assist with assigning PI-RADSv2 DWI category 4 or 5.

Project description:ObjectivesTo investigate the intra- and inter-rater reliability of the total radiomics quality score (RQS) and the reproducibility of individual RQS items' score in a large multireader study.MethodsNine raters with different backgrounds were randomly assigned to three groups based on their proficiency with RQS utilization: Groups 1 and 2 represented the inter-rater reliability groups with or without prior training in RQS, respectively; group 3 represented the intra-rater reliability group. Thirty-three original research papers on radiomics were evaluated by raters of groups 1 and 2. Of the 33 papers, 17 were evaluated twice with an interval of 1 month by raters of group 3. Intraclass coefficient (ICC) for continuous variables, and Fleiss' and Cohen's kappa (k) statistics for categorical variables were used.ResultsThe inter-rater reliability was poor to moderate for total RQS (ICC 0.30-055, p < 0.001) and very low to good for item's reproducibility (k - 0.12 to 0.75) within groups 1 and 2 for both inexperienced and experienced raters. The intra-rater reliability for total RQS was moderate for the less experienced rater (ICC 0.522, p = 0.009), whereas experienced raters showed excellent intra-rater reliability (ICC 0.91-0.99, p < 0.001) between the first and second read. Intra-rater reliability on RQS items' score reproducibility was higher and most of the items had moderate to good intra-rater reliability (k - 0.40 to 1).ConclusionsReproducibility of the total RQS and the score of individual RQS items is low. There is a need for a robust and reproducible assessment method to assess the quality of radiomics research.Clinical relevance statementThere is a need for reproducible scoring systems to improve quality of radiomics research and consecutively close the translational gap between research and clinical implementation.Key points• Radiomics quality score has been widely used for the evaluation of radiomics studies. • Although the intra-rater reliability was moderate to excellent, intra- and inter-rater reliability of total score and point-by-point scores were low with radiomics quality score. • A robust, easy-to-use scoring system is needed for the evaluation of radiomics research.

Project description:Purpose To determine the interobserver reproducibility of the Prostate Imaging Reporting and Data System (PI-RADS) version 2 lexicon. Materials and Methods This retrospective HIPAA-compliant study was institutional review board-approved. Six radiologists from six separate institutions, all experienced in prostate magnetic resonance (MR) imaging, assessed prostate MR imaging examinations performed at a single center by using the PI-RADS lexicon. Readers were provided screen captures that denoted the location of one specific lesion per case. Analysis entailed two sessions (40 and 80 examinations per session) and an intersession training period for individualized feedback and group discussion. Percent agreement (fraction of pairwise reader combinations with concordant readings) was compared between sessions. κ coefficients were computed. Results No substantial difference in interobserver agreement was observed between sessions, and the sessions were subsequently pooled. Agreement for PI-RADS score of 4 or greater was 0.593 in peripheral zone (PZ) and 0.509 in transition zone (TZ). In PZ, reproducibility was moderate to substantial for features related to diffusion-weighted imaging (κ = 0.535-0.619); fair to moderate for features related to dynamic contrast material-enhanced (DCE) imaging (κ = 0.266-0.439); and fair for definite extraprostatic extension on T2-weighted images (κ = 0.289). In TZ, reproducibility for features related to lesion texture and margins on T2-weighted images ranged from 0.136 (moderately hypointense) to 0.529 (encapsulation). Among 63 lesions that underwent targeted biopsy, classification as PI-RADS score of 4 or greater by a majority of readers yielded tumor with a Gleason score of 3+4 or greater in 45.9% (17 of 37), without missing any tumor with a Gleason score of 3+4 or greater. Conclusion Experienced radiologists achieved moderate reproducibility for PI-RADS version 2, and neither required nor benefitted from a training session. Agreement tended to be better in PZ than TZ, although was weak for DCE in PZ. The findings may help guide future PI-RADS lexicon updates. (©) RSNA, 2016 Online supplemental material is available for this article.

Project description:ObjectivesA watch and wait strategy with the goal of organ preservation is an emerging treatment paradigm for rectal cancer following neoadjuvant treatment. However, the selection of appropriate patients remains a challenge. Most previous efforts to measure the accuracy of MRI in assessing rectal cancer response used a small number of radiologists and did not report variability among them.MethodsTwelve radiologists from 8 institutions assessed baseline and restaging MRI scans of 39 patients. The participating radiologists were asked to assess MRI features and to categorize the overall response as complete or incomplete. The reference standard was pathological complete response or a sustained clinical response for > 2 years.ResultsWe measured the accuracy and described the interobserver variability of interpretation of rectal cancer response between radiologists at different medical centers. Overall accuracy was 64%, with a sensitivity of 65% for detecting complete response and specificity of 63% for detecting residual tumor. Interpretation of the overall response was more accurate than the interpretation of any individual feature. Variability of interpretation was dependent on the patient and imaging feature investigated. In general, variability and accuracy were inversely correlated.ConclusionsMRI-based evaluation of response at restaging is insufficiently accurate and has substantial variability of interpretation. Although some patients' response to neoadjuvant treatment on MRI may be easily recognizable, as seen by high accuracy and low variability, that is not the case for most patients.Key points• The overall accuracy of MRI-based response assessment is low and radiologists differed in their interpretation of key imaging features. • Some patients' scans were interpreted with high accuracy and low variability, suggesting that these patients' pattern of response is easier to interpret. • The most accurate assessments were those of the overall response, which took into consideration both T2W and DWI sequences and the assessment of both the primary tumor and the lymph nodes.

Project description:BackgroundThe Commission of the European Communities has published guidelines to be used as a gold standard for quality assessment of diagnostic radiographic images. Image quality and radiation dose must be monitored and optimally balanced for diagnostic purposes on patients. The objective of the current study was to assess intra- and inter-observer reproducibility in less experienced observers using the proposed European Guidelines on Quality Criteria for Diagnostic Radiographic Images in a quality assessment of lumbar spine radiographs in primary chiropractic practice in Denmark.MethodsTwo observers initially evaluated lumbar spine radiographs randomly selected from fifty chiropractic clinics, all connected to the national PACS server (KirPACS) in Denmark. All evaluations were performed twice by both observers using a four-week interval and for compliance with the European Quality Criteria for Diagnostic Radiographic Images. Inter- and intra-observer reproducibility was calculated using kappa statistics. In the interpretation of the kappa coefficient, the standards for strength of agreement reported by Landis and Koch were followed.ResultsThe strength of the inter-observer agreement of general image quality at baseline ranged from moderate agreement (k = 0.47) to substantial agreement (k = 0.68). After four weeks, the inter-observer agreement still ranged from moderate agreement (k = 0.59) to substantial agreement (k = 0.71), but with increased agreement for both kappa coefficients. In relation to intra-observer agreement of general image quality, the strength for observer A ranged from moderate (k = 0.58) to substantial (k = 0.72) and the strength for observer B overall was substantial (k = 0.63-0.75).ConclusionThe European Guidelines on Quality Criteria for Diagnostic Radiographic Images are considered a gold-standard and used in a method for quality assurance within the Danish chiropractic profession. The inter-rater and intra-rater agreements in this study, using the CEC-criteria, were found mostly acceptable. With appropriate attention to clear understanding of the individual criteria and sufficient training, this method is found to be reliable, even using less experienced observers, to carry out Diagnostic Radiographic Image Quality-assurance in primary care settings.

Project description:The application of toxicogenomics as a predictive tool for chemical risk assessment has been under evaluation by the toxicology community for more than a decade. However, toxicogenomics predominately remains a tool for investigative research rather than for regulatory risk assessment. In this study, we aimed to determine whether the current generation of microarray technology in combination with an in vitro experimental design was capable of generating robust, reproducible data of sufficient quality to show promise as a tool for regulatory risk assessment. To this end, we designed a prospective collaborative study to determine the level of inter- and intra-laboratory reproducibility between three independent laboratories. All test sites adopted the same protocols for all aspects of the toxicogenomic experiment including cell culture, chemical exposure, RNA extraction, microarray data generation and analysis. As a case study, the genotoxic carcinogen B[a]P and the human hepatocyte cell line HepG2 were used to generate three comparable toxicogenomic data sets. High levels of technical reproducibility were demonstrated using a widely employed gene expression microarray platform. While differences at the global transcriptome level were still observed between the test sites, a common subset of B[a]P responsive genes was identified across all test sites which included both genes previously reported in the literature as B[a]P responsive in addition to the same most highly up and down regulated genes. Pending further analysis, these preliminary data show promise that the current generation of microarray technology in combination with a standard in vitro experimental design can produce robust data that can be reproducibly generated in independent laboratories. Future work will need to determine whether in vitro model(s) can be not only reproducible but also predictive for a range of toxic chemicals with different mechanisms of action. Such an approach may potentially be part of future in vitro testing regimes for regulatory risk assessment.

Dataset Information

Intra- and interreader reproducibility of PI-RADSv2: A multireader study.

Publications

Intra- and interreader reproducibility of PI-RADSv2: A multireader study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets