Dataset Information

Reproducibility and Feasibility of Strategies for Morphologic Assessment of Renal Biopsies Using the Nephrotic Syndrome Study Network Digital Pathology Scoring System.

ABSTRACT: Context Testing reproducibility is critical for the development of methodologies for morphologic assessment. Our previous study using the descriptor-based Nephrotic Syndrome Study Network Digital Pathology Scoring System (NDPSS) on glomerular images revealed variable reproducibility. Objective To test reproducibility and feasibility of alternative scoring strategies for digital morphologic assessment of glomeruli and explore use of alternative agreement statistics. Design The original NDPSS was modified (NDPSS1 and NDPSS2) to evaluate (1) independent scoring of each individual biopsy level, (2) use of continuous measures, (3) groupings of individual descriptors into classes and subclasses prior to scoring, and (4) indication of pathologists' confidence/uncertainty for any given score. Three and 5 pathologists scored 157 and 79 glomeruli using the NDPSS1 and NDPSS2, respectively. Agreement was tested using conventional (Cohen ?) and alternative (Gwet agreement coefficient 1 [AC1]) agreement statistics and compared with previously published data (original NDPSS). Results Overall, pathologists' uncertainty was low, favoring application of the Gwet AC1. Greater agreement was achieved using the Gwet AC1 compared with the Cohen ? across all scoring methodologies. Mean (standard deviation) differences in agreement estimates using the NDPSS1 and NDPSS2 compared with the single-level original NDPSS were -0.09 (0.17) and -0.17 (0.17), respectively. Using the Gwet AC1, 79% of the original NDPSS descriptors had good or excellent agreement. Pathologist feedback indicated the NDPSS1 and NDPSS2 were time-consuming. Conclusions The NDPSS1 and NDPSS2 increased pathologists' scoring burden without improving reproducibility. Use of alternative agreement statistics was strongly supported. We suggest using the original NDPSS on whole slide images for glomerular morphology assessment and for guiding future automated technologies.

SUBMITTER: Zee J

PROVIDER: S-EPMC5946059 | biostudies-literature | 2018 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Reproducibility and Feasibility of Strategies for Morphologic Assessment of Renal Biopsies Using the Nephrotic Syndrome Study Network Digital Pathology Scoring System.

Zee Jarcy J Hodgin Jeffrey B JB Mariani Laura H LH Gaut Joseph P JP Palmer Matthew B MB Bagnasco Serena M SM Rosenberg Avi Z AZ Hewitt Stephen M SM Holzman Lawrence B LB Gillespie Brenda W BW Barisoni Laura L

Archives of pathology & laboratory medicine 20180219 5

Context Testing reproducibility is critical for the development of methodologies for morphologic assessment. Our previous study using the descriptor-based Nephrotic Syndrome Study Network Digital Pathology Scoring System (NDPSS) on glomerular images revealed variable reproducibility. Objective To test reproducibility and feasibility of alternative scoring strategies for digital morphologic assessment of glomeruli and explore use of alternative agreement statistics. Design The original NDPSS was ...[more]

PMID: 29457738

Similar Datasets

Project description:For a method to be widely adopted in medical research or clinical practice, it needs to be reproducible so that clinicians and regulators can have confidence in its use. Machine learning and deep learning have a particular set of challenges around reproducibility. Small differences in the settings or the data used for training a model can lead to large differences in the outcomes of experiments. In this work, three top-performing algorithms from the Camelyon grand challenges are reproduced using only information presented in the associated papers and the results are then compared to those reported. Seemingly minor details were found to be critical to performance and yet their importance is difficult to appreciate until the actual reproduction is attempted. We observed that authors generally describe the key technical aspects of their models well but fail to maintain the same reporting standards when it comes to data preprocessing which is essential to reproducibility. As an important contribution of the present study and its findings, we introduce a reproducibility checklist that tabulates information that needs to be reported in histopathology ML-based work in order to make it reproducible. Author summary For a method to be used a lot in medical research or clinical practice, it needs to be able to be reproduced so that people can trust it. Machine learning and deep learning have some challenges around this. For example, small changes in how a model is trained can lead to significant changes in the results of experiments. This makes it essential that researchers report how they do things in enough detail for the results of their experiments to be reproducible. In this work, we looked at three different algorithms used for digital pathology image analysis. We tried to reproduce them using only the information reported in their papers. We confirmed that even minor details could be essential. Authors often do not report all the details needed to reproduce their work. We also created a checklist of things that need to be reported to help other researchers make their work reproducible.

Project description:Background and objectivesUnfavorable histology on procurement biopsies is the most common reason for deceased donor kidney discard. We sought to assess the reproducibility of procurement biopsy findings.Design, setting, participants, & measurementsWe compiled a continuous cohort of deceased donor kidneys transplanted at our institution from 1/1/2006 to 12/31/2016 that had at least one procurement biopsy performed, and excluded cases with missing biopsy reports and those used in multiorgan transplants. Suboptimal histology was defined as the presence of advanced sclerosis in greater than or equal to one biopsy compartment (glomeruli, tubules/interstitium, vessels). We calculated κ coefficients to assess agreement in optimal versus suboptimal classification between sequential biopsy reports for kidneys that underwent multiple procurement biopsies and used time-to-event analysis to evaluate the association between first versus second biopsies and patient and allograft survival.ResultsOf the 1011 kidneys included in our cohort, 606 (60%) had multiple procurement biopsies; 98% had first biopsy performed at another organ procurement organization and their second biopsy performed locally. Categorical agreement was highest for vascular disease (κ=0.17) followed by interstitial fibrosis and tubular atrophy (κ=0.12) and glomerulosclerosis (κ=0.12). Overall histologic agreement (optimal versus suboptimal) was κ=0.15. First biopsy histology had no association with allograft survival in unadjusted or adjusted analyses. However, second biopsy optimal histology was associated with a higher probability of death-censored allograft survival, even after adjusting for donor and recipient factors (adjusted hazard ratio, 0.50; 95% confidence interval, 0.34 to 0.75; P=0.001).ConclusionsDeceased donor kidneys that underwent multiple procurement biopsies often displayed substantial differences in histologic categorization in sequential biopsies, and there was no association between first biopsy findings and post-transplant outcomes.

Project description:The multicenter Nephrotic Syndrome Study Network (NEPTUNE) digital pathology scoring system employs a novel and comprehensive methodology to document pathologic features from whole-slide images, immunofluorescence and ultrastructural digital images. To estimate inter- and intra-reader concordance of this descriptor-based approach, data from 12 pathologists (eight NEPTUNE and four non-NEPTUNE) with experience from training to 30 years were collected. A descriptor reference manual was generated and a webinar-based protocol for consensus/cross-training implemented. Intra-reader concordance for 51 glomerular descriptors was evaluated on jpeg images by seven NEPTUNE pathologists scoring 131 glomeruli three times (Tests I, II, and III), each test following a consensus webinar review. Inter-reader concordance of glomerular descriptors was evaluated in 315 glomeruli by all pathologists; interstitial fibrosis and tubular atrophy (244 cases, whole-slide images) and four ultrastructural podocyte descriptors (178 cases, jpeg images) were evaluated once by six and five pathologists, respectively. Cohen's kappa for inter-reader concordance for 48/51 glomerular descriptors with sufficient observations was moderate (0.40<kappa?0.60) for 17 and good (0.60<kappa?0.80) for 8, for 52% with moderate or better kappas. Clustering of glomerular descriptors based on similar pathologic features improved concordance. Concordance was independent of years of experience, and increased with webinar cross-training. Excellent concordance was achieved for interstitial fibrosis and tubular atrophy. Moderate-to-excellent concordance was achieved for all ultrastructural podocyte descriptors, with good-to-excellent concordance for descriptors commonly used in clinical practice, foot process effacement, and microvillous transformation. NEPTUNE digital pathology scoring system enables novel morphologic profiling of renal structures. For all histologic and ultrastructural descriptors tested with sufficient observations, moderate-to-excellent concordance was seen for 31/54 (57%). Descriptors not sufficiently represented will require further testing. This study proffers the NEPTUNE digital pathology scoring system as a model for standardization of renal biopsy interpretation extendable outside the NEPTUNE consortium, enabling international collaborations.

Dataset Information

Reproducibility and Feasibility of Strategies for Morphologic Assessment of Renal Biopsies Using the Nephrotic Syndrome Study Network Digital Pathology Scoring System.

Publications

Reproducibility and Feasibility of Strategies for Morphologic Assessment of Renal Biopsies Using the Nephrotic Syndrome Study Network Digital Pathology Scoring System.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets