Dataset Information

Hidden Variables in Deep Learning Digital Pathology and Their Potential to Cause Batch Effects: Prediction Model Study.

ABSTRACT:

Background

An increasing number of studies within digital pathology show the potential of artificial intelligence (AI) to diagnose cancer using histological whole slide images, which requires large and diverse data sets. While diversification may result in more generalizable AI-based systems, it can also introduce hidden variables. If neural networks are able to distinguish/learn hidden variables, these variables can introduce batch effects that compromise the accuracy of classification systems.

Objective

The objective of the study was to analyze the learnability of an exemplary selection of hidden variables (patient age, slide preparation date, slide origin, and scanner type) that are commonly found in whole slide image data sets in digital pathology and could create batch effects.

Methods

We trained four separate convolutional neural networks (CNNs) to learn four variables using a data set of digitized whole slide melanoma images from five different institutes. For robustness, each CNN training and evaluation run was repeated multiple times, and a variable was only considered learnable if the lower bound of the 95% confidence interval of its mean balanced accuracy was above 50.0%.

Results

A mean balanced accuracy above 50.0% was achieved for all four tasks, even when considering the lower bound of the 95% confidence interval. Performance between tasks showed wide variation, ranging from 56.1% (slide preparation date) to 100% (slide origin).

Conclusions

Because all of the analyzed hidden variables are learnable, they have the potential to create batch effects in dermatopathology data sets, which negatively affect AI-based classification systems. Practitioners should be aware of these and similar pitfalls when developing and evaluating such systems and address these and potentially other batch effect variables in their data sets through sufficient data set stratification.

SUBMITTER: Schmitt M

PROVIDER: S-EPMC7886613 | biostudies-literature | 2021 Feb

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Hidden Variables in Deep Learning Digital Pathology and Their Potential to Cause Batch Effects: Prediction Model Study.

Schmitt Max M Maron Roman Christoph RC Hekler Achim A Stenzinger Albrecht A Hauschild Axel A Weichenthal Michael M Tiemann Markus M Krahl Dieter D Kutzner Heinz H Utikal Jochen Sven JS Haferkamp Sebastian S Kather Jakob Nikolas JN Klauschen Frederick F Krieghoff-Henning Eva E Fröhling Stefan S von Kalle Christof C Brinker Titus Josef TJ

Journal of medical Internet research 20210202 2

<h4>Background</h4>An increasing number of studies within digital pathology show the potential of artificial intelligence (AI) to diagnose cancer using histological whole slide images, which requires large and diverse data sets. While diversification may result in more generalizable AI-based systems, it can also introduce hidden variables. If neural networks are able to distinguish/learn hidden variables, these variables can introduce batch effects that compromise the accuracy of classification ...[more]

PMID: 33528370

Similar Datasets

Project description:For a method to be widely adopted in medical research or clinical practice, it needs to be reproducible so that clinicians and regulators can have confidence in its use. Machine learning and deep learning have a particular set of challenges around reproducibility. Small differences in the settings or the data used for training a model can lead to large differences in the outcomes of experiments. In this work, three top-performing algorithms from the Camelyon grand challenges are reproduced using only information presented in the associated papers and the results are then compared to those reported. Seemingly minor details were found to be critical to performance and yet their importance is difficult to appreciate until the actual reproduction is attempted. We observed that authors generally describe the key technical aspects of their models well but fail to maintain the same reporting standards when it comes to data preprocessing which is essential to reproducibility. As an important contribution of the present study and its findings, we introduce a reproducibility checklist that tabulates information that needs to be reported in histopathology ML-based work in order to make it reproducible. Author summary For a method to be used a lot in medical research or clinical practice, it needs to be able to be reproduced so that people can trust it. Machine learning and deep learning have some challenges around this. For example, small changes in how a model is trained can lead to significant changes in the results of experiments. This makes it essential that researchers report how they do things in enough detail for the results of their experiments to be reproducible. In this work, we looked at three different algorithms used for digital pathology image analysis. We tried to reproduce them using only the information reported in their papers. We confirmed that even minor details could be essential. Authors often do not report all the details needed to reproduce their work. We also created a checklist of things that need to be reported to help other researchers make their work reproducible.

Dataset Information

Hidden Variables in Deep Learning Digital Pathology and Their Potential to Cause Batch Effects: Prediction Model Study.

Background

Objective

Methods

Results

Conclusions

Publications

Hidden Variables in Deep Learning Digital Pathology and Their Potential to Cause Batch Effects: Prediction Model Study.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets